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Deborah La Torre Matrundola, Sarah Reber, and Kilchan Choi 
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Abstract 

The Bill and Melinda Gates Foundation invested in the Literacy Design Collaborative (LDC) 
as one strategy to support teachers’ and students’ transition to the Common Core State 
Standards (CCSS) in English language arts. This report provides an early look at the 
implementation of LDC in sixth-grade Advanced Reading classes in a large Florida district, 
and the effectiveness of the intervention in this setting. The study found that teachers 
understood LDC and implemented it with fidelity and that curriculum modules were well 
crafted. Teachers also generally reported positive attitudes about the effectiveness of LDC 
and its usefulness as a tool for teaching CCSS skills. Although implementation results were 
highly positive, quasi-experimental analyses employing matched control group and 
regression discontinuity designs found no evidence of an impact of LDC on student 
performance on state reading or district writing assessments. Furthermore, students generally 
performed at basic levels on assessments designed to align with the intervention, suggesting 
the challenge of meeting CCSS expectations. Exploratory analyses suggest that LDC may 
have been most effective for higher achieving students. However understandable, the findings 
thus suggest that, in the absence of additional scaffolding and supports for low-achieving 
students, LDC may be gap enhancing. 

Chapter 1: Introduction 

The Common Core State Standards (CCSS) in English language arts (ELA) bring rigorous, 
new demands for student learning to ensure that students will have the literacy knowledge and 
skills they need to be prepared for success in college and careers. For most schools, these new 
English language arts standards dramatically increase expectations for students’ ability to read 
literary and informational texts closely, analyze evidence, communicate orally and in writing for 
a variety of audiences and purposes, and conduct research. 

The new standards bring new challenges for teachers and students alike. Teachers must 
adapt their existing curriculum and instruction to align with the new standards and adopt new 
pedagogical approaches to support their students’ success. Recognizing these needs, the Bill and 
Melinda Gates Foundation invested in the Literacy Design Collaborative (LDC) as one strategy 
to support teachers’ and students’ transition to these new, more rigorous expectations. Although 
LDC is at a relatively early stage of implementation, the Foundation was interested in getting an 
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early read on program effectiveness and contracted with the National Center for Research on 
Evaluation, Standards, and Student Testing (CRESST) to conduct two quasi-experimental studies 
of LDC’s implementation and learning impact. The study reported here examines LDC as it was 
implemented in sixth-grade Advanced Reading courses in a major urban school district in 
Florida during the 2012-2013 school year. The second study, examining LDC implementation in 
eighth-grade history/social studies and science classes during the 2012-2013 school year in 
selected Kentucky and Pennsylvania school districts is reported in a separate companion report 
(Herman et ah, 2015). 

This chapter presents background on the study, including a brief description of the LDC 
intervention and the evaluation questions that guide the Florida study. In the following chapters, 
we summarize study methodology, present implementation and outcome results, and examine the 
implications of our findings. 

Literacy Design Collaborative Overview 

LDC supports the transition to the Common Core State Standards in English language arts 
by providing flexible module templates that enable middle and high school teachers and schools 
to integrate reading, research, and writing standards into instruction. End-of-module, extended 
writing tasks provide the heart of the approach. Teachers use fill-in-the-blank templates to design 
a culminating content-focused writing task, which then is used to organize a module of 
instruction. The module is designed to address relevant content in literature, history/social 
studies, or science as well as relevant reading and writing demands aligned with the CCSS. For 
example, the following templates structure end-of-module tasks for students’ argumentative and 
expository writing respectively: 

TASK 1 TEMPLATE (Argumentative/Analysis LI, L2, L3): After researching 

(informational texts) on (content), write an (essay or substitute) that argues 

your position, pro or con, on (content). Support your position with evidence from 

your research. L2. Be sure to acknowledge competing views. L3. Give examples from past or 
current events or issues to illustrate, clarify, and support your position. (Appropriate for: 
social studies, science) 

TASK 11 TEMPLATE (Informational or Explanatory/Definition El, L2): After researching 

(informational texts) on (content), write a (report or substitute) 

that defines and explains (content). Support your discussion with evidence from 

your research. L2. What implications can you draw? (Appropriate for: ELA, social studies, 
science ) 

After deciding on the end-of-module writing task, teachers then use an LDC-specified 
framework (or instructional ladder ) to design instructional activities to support students in 
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developing the content and requisite literacy skills to successfully complete the culminating task. 
The steps of the ladder include core activities, such as note-taking, identifying evidence to 
support claims, and evaluating contrasting positions, that scaffold student learning and provide 
ongoing opportunities for formative assessment. The final product — instructional ladder plus 
template task — is referred to as an LDC module. 

The Foundation has been exploring a variety of approaches and partners to support LDC 
implementation. The approaches vary in the extent of professional development and coaching 
support that teachers and schools receive and in their focus on individual teachers or districtwide 
and/or schoolwide implementation. The Florida study focuses on a centralized, districtwide LDC 
implementation. 

The Florida District Context 

Among the 10 largest districts in the country, the study site provides an interesting case for 
examining how LDC fares in a highly diverse student population. As described for the study year 
by district records, student demographics were approximately as follows: 

• Ethnicity: African American, 22%; Hispanic, 31%; White, 39%; Other, 9% 

• Socioeconomic status: 59% qualify for free or reduced price lunch 

• Language status: 12% English language learners 

• Achievement level: 63% of sixth-graders at or above proficient in reading 

Relative to the rest of the state, the study district is more diverse in ethnicity and proportion 
of English language learners and slightly lower in socioeconomic status and in student 
achievement than the state average. Students placed in Advanced Reading, the subject of this 
study, were on average higher performing than the mean student in the state based on prior year 
reading scores. 

This Florida district took a unique, centralized approach to LDC implementation in sixth- 
grade Advanced Reading courses. District literacy leaders in collaboration with the on-site 
reading coaches from 10 pilot schools initially developed four LDC modules to make up the core 
of the sixth-grade Advanced Reading curriculum. The modules were piloted in 2010-2011 at the 
10 schools and revised in collaboration with teachers during summer professional development 
sessions. In the following year, an introductory module was added to the suite and 
implementation was expanded districtwide to all 46 district middle schools. Teachers were 
expected to implement the five modules according to a districtwide pacing schedule and were 
provided detailed plans for the instructional activities and culminating performances for each of 
the five modules: 
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• Introduction to LDC, which provides an orientation; 

• Personal Change Creates Community, in which students produce an informational 
essay that focuses on how and why young people make a difference in their 
communities; 

• Savage Harvest, in which students write an article for a children’s magazine that 
defines child labor, explains child labor practices and how children around the world 
are impacted, and considers consumer issues; 

• Communication in a Cybernation, in which students write an editorial that defines our 
digital footprints and explains their perspective on digital privacy; and 

• Fit for Life, in which students write a public service announcement that explains the 
long-term benefits of regular physical activity. 

Each module includes step-by-step guidance for teaching the module, including (a) an 
overview of the module goals and the standards it addresses; (b) a pacing guide that summarizes 
the content of each lesson and a grading matrix indicating what parts of each lesson may be 
graded for participation/classwork, writing, assessment, or other; (c) exit slips to be used with 
specified lessons; (d) lesson plans for each lesson, including a task analysis of lesson goals and 
timing of specific activities that can be used to achieve the goals, with instruction notes, 
assessment activities, and needed handouts; (e) culminating teaching and assessment tasks for 
the module; (f) possible read alouds; and (g) tie-ins to Florida Comprehensive Achievement Test 
(FCAT) objectives. 

Evaluation Questions 

At the time of the study, sixth-grade Advanced Reading teachers in the pilot schools had 
two years of prior experience in implementing LDC, while the vast majority of study teachers in 
the remaining 36 middle schools had only one prior year of experience. At this early phase of 
LDC development, the study addresses a comprehensive set of evaluation questions: 

• How do teachers implement LDC? 

• What is the impact of LDC on student learning? 

• What conditions and contexts, including quality of implementation, influence LDC 
effectiveness? 

In addressing these questions, the study implemented a quasi-experimental design and 
developed and validated new measures of implementation and learning impact, as described in 
the next chapter. 
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Chapter 2: Study Methodology 


The study focused on sixth-grade teachers and students in Advanced Reading during the 
2012-2013 school year. Study methodology featured strong quasi-experimental designs (QED) 
to examine LDC’s effects on students’ state assessment performance and local district measures, 
as well as implementation and outcome measures that were specially developed to align well 
with LDC goals. The implementation measures included teacher logs, an end-of-year survey, and 
analysis of module quality. Below we provide more detail on these elements of study 
methodology. 

Study Sample 

Population. The study population included all sixth-grade Advanced Reading teachers and 
their students in the district during the 2012-2013 school year who had at least one year of prior 
experience using LDC. This population included teachers from 41 schools including eight of the 
10 Phase 1 schools (not all of the 46 Phase 1 and Phase 2 schools had teachers eligible for the 
study). All sixth-grade Advanced Reading teachers were required to implement the five LDC 
modules according to the same sequence, and as mentioned, these modules constituted the bulk 
of the yearlong curriculum. 

Sample for quasi-experimental analyses. As LDC was implemented districtwide in the 
study district, LDC treatment for the quasi-experimental analyses was based on school 
identifiers. For reasons described further in Chapter 4, certain analyses excluded Phase 1 schools 
(the earliest implementers). The matched control group analysis used longitudinal student and 
teacher data from Florida’s state database. Drawing on sixth-grade students in similar courses 
across the state, we used propensity matching techniques to create a comparison sample of 
students who were equivalent to the LDC group in demographics and prior academic 
performance. The regression discontinuity design compared students close to the cut-point for 
entry into Advanced Reading who were selected and not selected into the course. These 
techniques and the resulting samples are described further in Chapter 4 in the context of the QED 
results. 

Table 1 shows the demographic and entering achievement characteristics of the full 
population of LDC students in the study, based on district records. These data indicate that the 
study’s LDC student population was ethnically diverse, and more than half the students qualified 
for free or reduced price lunch. Proportions of students classified as English language learners or 
special education were small and lower than district means (not shown), which is not surprising 
given that the study population is composed of students in Advanced Reading classes. 
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Table 1 


Demographic Characteristics ofLDC Student Population 
All Sixth-Grade Advanced Reading Students (n = 6926) 


Student characteristics 

Value 

Ethnicity 


Hispanic (%) 

26.1 

White (%) 

47.8 

African American (%) 

14.7 

Asian (%) 

4.7 

Other (%) 

6.6 

Qualify for free or reduced price lunch (%) 

51.4 

English language learner (%) 

1.3 

Gender: Female (%) 

52.3 

Special education (%) 

4.2 

Mean prior FCAT reading performance 

233.7 


Sample recruitment and completion rates for LDC-only measures. District leadership 
and LDC leaders and coordinators fully supported study recruitment, but teacher participation in 
the special measures was voluntary. The effective study sample size thus varied with the various 
instruments used in the study. Table 2 displays the number of study-eligible teachers who were 
invited to participate in the study, those who agreed to participate, and the completion rates for 
each of the study measures. Because no special agreement was needed to include teachers in the 
analysis of available state and local assessment and demographic data, all teachers in the study 
population were included in the QED study sample. 

Nearly 60% of the eligible Advanced Reading teachers agreed to participate in the study 
and half or more completed the major research activities, including logging about modules, 
completing the end-of-year survey, and administering the specially developed student learning 
measure, the Integrated Learning Assessment (ILA). 
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Table 2 


LDC Study Completion Rates by Instrument 


Sixth-grade Advanced Reading teachers 

n 

% relative to all eligible 
LDC teachers 

% relative to 
consented teachers 

Teachers eligible for CRESST study 

101 

-- 

- 

Teachers consenting to CRESST study 

59 

58 

- 

Completed logs 

52 

51 

88 

Administered ILA measure 

53 

52 

90 

Completed survey 

56 

55 

-- 


Based on students who took the ILA, Table 3 indicates that the students represented in the 
special measures sample are very similar in student characteristics and entering achievement to 
the full Advanced Reading student population (as shown in Table 1). 

Table 3 


Demographic Characteristics of LDC Student Sample: 
Advanced Reading Students Completing Special Measures 
(n = 877) 


Student characteristics 

Value 

Ethnicity 


Hispanic (%) 

29.4 

White (%) 

44.9 

African American (%) 

14.4 

Asian (%) 

4.9 

Other (%) 

6.2 

Qualify for free or reduced price lunch (%) 

55.7 

English language learner (%) 

3.7 

Gender: Female (%) 

52.3 

Special education (%) 

1.3 

Mean prior FCAT reading performance 

233.1 


Implementation Measures 

Our implementation measures draw on research on instruction and instructional change, 
given that the ultimate goal of the LDC intervention is to align teachers’ instruction to Common 
Core State Standards. Classroom practice is notoriously impervious to reform (Cuban, 1984; 
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Lortie, 1975), however, an emerging body of research has documented the relationship between 
student achievement and specific instructional practices that create opportunities to learn (see 
Bryk, Sebring, Allenworth, Luppescu, & Easton, 2010; Rowan & Correnti, 2009; Winters & 
Herman, 2011). Our implementation measures thus focus on classroom instruction, while 
recognizing that multiple factors influence and inhibit teacher innovation and instructional 
change. Study measures include web-based teacher logs, collected twice weekly during LDC 
module implementation, and an end-of-year teacher survey. For both measures, teachers who 
taught multiple sections of sixth-grade Advanced Reading were asked to focus on the same, 
randomly selected focal class. 

Web-based teacher log. Study teachers were asked to complete a log twice weekly during 
their implementation for each of two LDC modules: Module 3, Savage Harvest (which focuses 
on international child labor practices), and Module 4, Communication in a Cybernation. The 
Savage Harvest and Communication in a Cybernation modules were typically completed 
between November and January and between January and April respectively. The logs focused 
on (a) the degree to which instruction generally aligned with the structure of the LDC 
intervention, (b) the degree to which instruction explicitly specified and addressed the discrete 
literacy skills required to complete the summative writing task, and (c) the quality and extent of 
formative assessment practices incorporated into LDC instruction. Each log was designed to 
capture classroom instruction on the particular day the log was completed and focused on only 
one of the teacher’s classes — the same class for all logs. 

The log included opening (gateway) items that asked teachers to specify which component 
of the LDC module they addressed on that particular day (i.e., Preparing for the Task, Reading 
Process, Transition to Writing, Writing Process) and then branched to back-end items for the 
identified component(s), where teachers answered additional questions about component 
instructional objectives and strategies. For example, the Reading Process section asked teachers 
to check all the specific reading skills that were addressed in the day’s instruction. Follow-up 
items asked teachers to identify how they assessed student understanding and/or reading skills 
during the period and how they responded if a student had difficulty with the reading assignment. 
The emphasis on formative assessment aligns with LDC intent, which views the steps in the 
instructional ladder as opportunities for the teacher to track student progress and intervene 
appropriately to support student learning. Logs were analyzed at the teacher level. 

On average, participating teachers submitted 12 logs for Savage Harvest and 11 for 
Cybernation, for a total of 23 logs per teacher. Log data were summarized across logs for each 
teacher by module. The computations included teachers’ responses only to those items that were 
associated with the LDC component(s) that the teacher specified for each log. For example, log 



reading component mean scores were based only on responses to logs for which teachers 
reported implementing the reading component of the module. (See Exhibit A1 in Appendix A for 
a copy of the log.) Group means were then computed across teachers and modules. 
Implementation analyses also considered various composite measures, as described further in 
implementation results below. Data were analyzed for each module separately, but because few 
consistent differences were found, the report summarizes results across the two modules. 

Teacher surveys. CRESST collaborated with Research for Action (RFA) on the design of 
a 2013 implementation and scale-up survey for teachers. The survey included a section on 
module implementation with items designed to mirror the intent of the log items. These survey 
items queried 

• relative time spent on the various module components; 

• relative emphasis given to specific reading and writing skills; 

• use of formative assessment and strategies for providing feedback; and 

• perceptions of LDC impact. 

Further, we drew on RFA survey variables as context and possible moderators of LDC 
implementation and impact — for example, experience using LDC, attitudes regarding literacy 
instruction, extent of professional development, leadership support, and collaboration. 
Descriptive statistics were computed at the teacher level. (See Exhibit A2 in Appendix A for 
copy of the LDC teacher survey.) 

Analysis of LDC modules. District notebooks, providing guidance and student activities 
for each module, were analyzed using the specially developed CRESST Assignment Measure 
rubric. The rubric addresses nine dimensions of quality, which focus on the quality of the central 
writing task, the quality of the instructional ladder, and the overall coherence and quality of the 
module, and attend to both literacy and content demands: 

• Dimension 1: Effective Writing Task 

• Dimension 2: Alignment to the CCSS and Local and State Literacy and Content 
Standards 

• Dimension 3: Text Alignment 

• Dimension 4: Text Appropriateness 

• Dimension 5: Text Rigor 

• Dimension 6: Fidelity to LDC Module Instruction 

• Dimension 7: Quality Instructional Strategies 

• Dimension 8: Coherence and Clarity of Module 
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• Dimension 9: Overall Impression 

Each dimension was scored on a 1-5 scale, where a score of 1 indicated that the quality 
was absent and/or there was not apparent attention to the dimension, a score of 3 indicated the 
quality was moderately realized, and a score of 5 indicated that the quality of the dimension was 
fully realized. (See Exhibit A3 in Appendix A for the scoring rubric.) Detailed rubrics describe 
each dimension and the criteria defining each score value. 

A companion study examined the use of the rubric in analyzing LDC units in eighth-grade 
history/social studies and science. This study examines the measurement quality of module 
scores through generalizability, factor analysis, and decision study methodology and is reported 
as a separate paper. (See Reisman, Herman, Luskin, & Epstein, 2013, in Appendix B, which 
describes the measures, including development, piloting, scoring, generalizability and 
dependability studies, and results.) Raters generally found the scoring dimensions intuitive and 
well aligned with the available artifact data. Both the social studies and science analyses revealed 
low rater variance and high teacher (or teacher by module) variation. Moreover, the factor 
analyses indicated that all dimensions load on a single factor, making the case that the CRESST 
Assignment Measure effectively measures a coherent trait that might be understood to be LDC 
implementation, or perhaps more generally, instructional quality in the integration of literacy and 
content. 

For this study, both a study researcher involved in rubric development and a middle school 
social studies teacher specially trained to apply the rubric rated each of the four modules. Any 
score discrepancies were addressed through consultation and consensus. 

Student Outcome Measures 

Student outcome measures for the study include state assessment data, local district data, 
and CRESST-developed Integrated Literacy Assessments (ILAs). Student demographic 
information also was secured with the available state assessment data. 

State assessment data. The study used developmental scale scores from the Florida 
Comprehensive Achievement Test (FCAT 2.0) to measure students’ 2012-2013 performance in 
English language arts, specifically reading. The FCAT 2.0 reading development scale scores are 
vertically aligned to track student longitudinal progress from year to year, from Grades 3 to 10. 
The sixth-grade FCAT 2.0 Reading Test is composed of 50-55 multiple choice items. According 
to test specifications (see http://www.fldoe.org/core/fileparse.php/5682/urlt/0077907- 
fll0spisg68rwtr3gfinal.pdf), the items on the sixth-grade reading test are allocated proportionally 
into the following categories: 

• Vocabulary: 20% 
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• Reading application: 30% 

• Literary analysis, fiction/non-fiction: 30% 

• Informational text: 20% 

Note that the allocation gives only 20% of the item to analysis of informational text, which 
tends to be an emphasis in LDC modules. 

Individual-level longitudinal data were obtained for all students statewide from 2008-2009 
to 2012-2013. However, an important limitation of these data is that FCAT moved to Version 
2.0 for the 2010-2011 school year, coinciding with the Phase 1 pilot of LDC. The change in tests 
was accompanied by a differential drop in performance for students in the study district’s 
Advanced Reading courses, relative to those in the rest of the state. Because Phase 2 schools that 
had not yet begun implementation of LDC also experienced the drop, we questioned the 
comparability of FCAT 1.0 and 2.0 for study purposes. Unable to disentangle LDC effects from 
the effects of the change in the test and lacking a comparable baseline measure for Phase 1 
schools, we made the decision to limit our matched control group analyses to the 30 Phase 2 
schools. For Phase 2 schools, FCAT 2.0 data were available from prior to LDC implementation 
through the study year. (We provide additional information on our rationale and approach in the 
chapter providing QED results.) 

District assessment data. CRESST additionally received data for students in the study 
district on the district’s writing assessment. 

The district’s writing test mirrors the Florida Writes assessment (which is part of the state 
assessment system for Grades 4, 8, and 10) for grades not assessed by the state and is 
administered at both the beginning and end of the year. In contrast to the writing-from-reading 
emphasis of LDC, the district and state writing assessments ask students to respond to a prompt 
that can be answered based solely on students’ prior knowledge. The prompts call on narrative, 
expository, or persuasive writing. For example, recent prompts for 2013-2014 Florida Writes for 
Grades 4 and 8 were: 

Grade 4: Suppose you won something special. 

Think about winning something special. 

Now write a story about what happened when you won something special. 


Grade 8: Suppose you could convince a famous person to visit your town. 
Think about why this person should visit your town. 

Now write to convince this person to visit your town. 
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Beginning- and end-of-year sixth-grade district writing scores were provided for 2009- 
2010 to 2012-2013. In 2010-2011 and 2011-2012, the district writing assessment used a six- 
point holistic rubric to score student essays, based on the rubric used for the state writing test in 
Grades 4, 8, and 10. The rubric integrated four elements in quality writing: focus, organization, 
support, and conventions. In 2012-2013, the district began using the LDC rubric to score 
students on its writing assessment. The LDC rubric is scored on a four-point scale from 1 to 4, 
and incorporates the following score elements: focus, controlling idea, reading/research, 
development, organization, conventions, and content understanding. 

Integrated Literacy Assessment (ILA). The CRESST ILAs are designed to measure both 
students’ literacy development relative to the CCSS in English language arts and the depth of 
students’ content understanding in literature, social studies/history, or science. Across content 
areas, the two-day ILAs feature a consistent structure that roughly mirrors components of LDC: 
On Day 1, students read several texts that typify those encountered in the discipline and address 
an important content principle or theme and respond to selected and constructed response 
reading comprehension and analysis questions about each text. The questions are aligned with 
the CCSS in ELA. On Day 2, students respond to an essay prompt that, consistent with the 
CCSS, asks them to synthesize what they know with what they have read to produce an 
evidence-based, extended explanation or argument responding to a content-related problem. 

ILA content foci. The ILA design seeks to respond to the challenge of disentangling the 
background information that students bring to the assessment from the knowledge that they 
gather from reading the actual texts in the assessment (Klein, 1983). Assessment designers and 
evaluators must take care to not unfairly privilege students whose teachers spent considerable 
time on a topic over those whose teachers’ coverage was more cursory. We addressed this 
potential confound in the ILAs in two ways. First, we selected topics that students should have 
covered in their recent curriculum, or that were closely related to topics they had covered. 
Second, we included relevant background knowledge in the actual exam so that even students 
with virtually no familiarity in the topic could orient themselves to the substance of the texts and 
write meaningfully about them (Baker, 1994). 

To ensure that students had an opportunity to learn the content topic, the ILA for the study 
drew on the content of one of the LDC modules, child labor. The assessment also includes 
background information in the fonn of three related texts. Each of the three documents in the 
assessment includes a headnote with key background infonnation about the author and the 
context. The ILA was designed specifically to align with the Savage Harvest module, the second 
of the four required modules for sixth-grade Advanced Reading, which focuses on contemporary 
international child labor. The intent was for students to use both their existing knowledge of the 
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topic and new information garnered from the readings in the ILA to successfully complete the 
final writing task. A copy of the Child Labor ILA can be found in Exhibit A4 in Appendix A. 

ILA administration and scoring. As described earlier, CRESST’s ILAs are divided into a 
reading component, in which the student reads and responds to a variety of texts through 
multiple choice and short extended response questions, and a final, extended writing task. The 
ILAs were scored using rubrics developed by the CRESST ILA development team. As outlined 
in Table 4, the writing task was scored on five dimensions: content understanding, rhetorical 
structure/quality, organization, reference support with text, and grammar and conventions. Each 
of these dimensions was rated on a four-point scale, with a 4 representing advanced perfonnance, 
3 representing proficiency, 2 representing a basic level of performance, and 1 below basic. Table 
5 presents an example of a scoring rubric for one of the four dimensions: rhetorical 
structure/quality. Scores on the five dimensions were summed to arrive at an overall score for the 
essay. Copies of the writing and short answer rubrics for the Child Labor ILA can be found in 
Exhibit A5 in Appendix A. 


Table 4 


Scoring Rubric for ILA Final Writing Tasks 


Dimension 

Name 

Description 

A 

Content understanding 

This is a measure of overall how well the student has demonstrated that they 
understand the materials and the topic in their essay. 

B 

Rhetorical 

structure/quality 

Argument: establishes a claim, acknowledging alternate or opposing claims, 
and supports it consistently with relevant evidence and logical reasons. 
Explanation: establishes a thesis; previews the main points; and thoroughly 
develops the topic with well-chosen information, examples, and analysis. 

C 

Organization 

Consistent focus, logical progression of ideas, and structure appropriate for 
the task. 

D 

Reference/support with 
text 

This is a measure of how well statements in the essay are supported by 
references to text details. A text detail is a quotation, paraphrase, or any other 
reference to information and ideas in the texts provided. 

E 

Grammar and 
conventions 

The essay is written with a command of standard English conventions: 
proper English usage and control of grammar, appropriate tone, paragraph, 
and sentence structure. 
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Table 5 

Scoring Dimension Example for Rhetorical Structure/Quality 


Description Score 

Important elements of the argument are clearly and thoroughly described and articulated. 4 
Elements of the argument are clearly described. 3 

There is an attempt to describe some elements of the argument. 2 

Elements of the argument are not described, or the descriptions are unclear. 1 


CRESST recruited a small group of classroom teachers to score the essays, most of whom 
had participated in scoring for an earlier validation study. The group met for two 6-hour training 
sessions. During training, raters practiced applying the five-dimension essay rubric until they 
achieved 80% reliability with criterion papers prescored by CRESST staff. Essays were divided 
evenly among the qualifying raters, with 20% of the essays assigned to two raters. In addition, 
about 10 of the essays interspersed throughout the scoring process were designated check papers 
and scored by all raters. Check papers were distributed so that raters would score them in the 
same order. CRESST staff monitored the scoring daily to ensure that raters were progressing and 
that check papers were being scored consistently. If a rater’s scores were inconsistent with check 
papers, their double-scored papers would be checked and feedback and additional training 
provided as needed. After scoring, interrater reliability was determined from the double-scored 
essays. One teacher was found not to have sufficient reliability and that rater’s scores were 
dropped. Essays scored by that rater were divided between the two most reliable remaining raters 
and rescored. 

Short answer items were scored on either a 0-1 or 0-2 point scale, with partial credit 
possible on the three-point scale. Rubrics were prepared for each item. The rubrics were based 
on those used in an earlier assessment validation study. 

Table 6 and Table 7 document rater agreement on the essay scores, including items that 
were double-scored and check papers scored by all three raters. The data show substantial rater 
agreement, with percentage of exact agreement obviously higher in the case of two raters relative 
to agreement among the three. In virtually all cases, there is 100% agreement plus or minus one 
score point. 
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Table 6 


Interrater Reliability for Double-Scored Child Labor ILA Essays (n = 104) 


Dimension 

% exact agreement 

% agreement within 1 point 

Content understanding 

73 

98 

Rhetorical structure/quality 

68 

100 

Organization 

76 

99 

Reference/support with text 

75 

99 

Grammar and conventions 

71 

100 


Table 7 


Interrater Reliability for Triple-Scored Child Labor ILA Essays (n = 16) 


Dimension 

% exact agreement 

% agreement within 1 point 

Content understanding 

63 

100 

Rhetorical structure/quality 

56 

100 

Organization 

94 

100 

Reference/support with text 

69 

94 

Grammar and conventions 

81 

100 


Table 8 presents the reliability of the reading and writing components of the Child Labor 
ILA. Cronbach’s alpha reliability statistic for the 13 items in the reading component of the 
assessment is .414, while the internal consistency of the five dimension scores for the writing 
component is .860. Note that these statistics should not be directly compared, as the writing 
dimension scores assess the same task and are not independent. The reading reliability is lower 
than expected and indicates substantial error in the scores. 

Table 8 


Reliability of Child Labor Integrated Learning Assessment 


Component 

Number of items/dimensions 

Cronbach’s alpha (reliability) 

Reading 

13 

.414 

Writing 

5 

.860 
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Chapter 3: LDC Implementation 


All sixth-grade Advanced Reading teachers in the study district were required to implement 
an introductory module, followed by four complete modules over the course of the school year: 
Personal Change Creates Community, Savage Harvest, Communication in a Cybernation, and Fit 
for Life, which collectively represented the bulk of the Advanced Reading curriculum for the 
year. In this chapter we review results that bear on how these modules were implemented, 
drawing on teacher logs and surveys, and an analysis of the modules themselves. In reviewing 
these findings, it is important to keep in mind the relatively small sample sizes and that while 
log, survey, and module samples overlap, they are not fully the same. For example, some 
teachers completed the survey but not the log and vice versa. 

Teacher Background 

Teachers’ background, prior experience, and attitudes about literacy instruction, gleaned 
from teacher survey responses, provide important context for the implementation findings. 
Teachers participating in LDC varied greatly in teaching experience, with both novice and 
seasoned veteran teachers in the sample. On average, teachers reported having been in the 
profession for almost 14 years, with the majority of teachers’ experience within the study district, 
and just under half in their current schools. Almost all teachers reported teaching students who 
were reading and/or writing below grade level as well as those who had advanced literacy skills, 
and the great majority were also teaching classes with English language learners and special 
education students (see Tables Cl and C2 in Appendix C). 

The vast majority of survey respondents reported that their participation in LDC was 
required (rather than voluntary), consistent with the district mandate to use the modules for sixth- 
grade Advanced Reading classes. Teachers reported teaching an average of 3.6 modules in 
2011-2012 and 4.3 modules in 2012-2013, a finding that confirms that the teachers we targeted 
for the study were experienced in implementing LDC. Not surprisingly, given that modules in 
the district were developed centrally, few teachers reported being involved with module 
development. Teachers generally agreed with a central LDC and CCSS premise that literacy 
instruction should be integrated into content area teaching, and that reading and writing are 
essential supports for students’ content area learning (see Tables C3, C4, and C5). 

Log Findings 

As noted earlier, teachers completed twice-weekly logs for the Savage Harvest and 
Communication in a Cybernation modules, focusing on one randomly selected Advanced 
Reading classroom. These data provide information on the forms of activities in which students 
were engaged during module implementation, the specific reading and/or writing activities in 
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which students participated, and teachers’ use of formative assessment. Note, however, that 
teachers’ responses showed wide variation, so reported means must be interpreted with caution. 

Teachers reported spending approximately 30 to 40 minutes a day on LDC instruction and, 
as Table D1 in Appendix D shows, this time included a wide range of activities. The data suggest 
substantial variation across teachers, but in general, teachers most frequently engaged students in 
independent reading or writing, which made up approximately one third of classroom time. 
Small group work constituted the second most frequent activity, at 13% to 14% of class time for 
the average teacher, and explicit strategy instruction and whole-class discussion constituted 
roughly 10% of classroom activity time. The remainder of class time was spread over a variety 
of different activity types. The time typically spent in explicit strategy instruction may be of 
special note, because it indicates that students received relatively little direct instruction in the 
skills they need to develop their reading and writing skills. Explicit strategy instruction, however, 
also is one of the only areas that appeared to show a substantial increase from Savage Harvest to 
Cybernation. 

Examining how the separate components of LDC were implemented, Table D2 shows how 
teachers transitioned to the LDC module and introduced the module tasks. Most commonly, 
teachers reviewed the content of the module and connected that content to what students had 
previously learned. Less frequently were students provided an advance organizer for the writing 
task they would be asked to complete or provided success criteria. 

To prepare for the module, students primarily listened to the teacher and sometimes were 
engaged with graphic organizers, discussed applicable strategies, made predictions about the 
topic or readings, or generated questions (see Table D3). 

During the LDC reading process component, students most frequently were engaged in 
independent reading, summarizing the main points of the reading, and to a lesser extent drawing 
conclusions from the evidence and note-taking. While other aspects of critical reading and 
analysis generally received less attention, there appeared to be an increase in the emphasis on 
these skills from Savage Harvest to Cybernation — for example, in distinguishing fact from 
opinion, evaluating the strength and weaknesses of evidence, comparing arguments, and 
examining the author’s perspective (see Table D4). 

To check students’ understanding and progress, teachers most typically circulated as 
students were working independently, listened as students discussed text with peers, and 
collected and reviewed student work. To a lesser extent, teachers reported asking students to 
respond to oral questions, leading whole-class discussions, or listening to students’ questions 
(see Table D5). Providing one-on-one feedback, asking a peer to help, and offering a hint or 
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suggestion were the most frequent strategies teachers used when they discovered 
misunderstandings about reading (see Table D6). 

Moving to the writing process component, the data suggest that teachers’ emphases moved 
from general to more specific aspects of writing from Savage Harvest to Cybernation. While the 
skills related to the basic structure of writing were emphasized across both modules, the specifics 
of effective writing got greater attention during Cybernation, for example, formulating a thesis 
statement and/or counterargument; writing an introduction, body paragraph, and conclusion; and 
incorporating relevant quotes and evidence from texts. The data suggest attention to a 
progression of student skills and are consistent with the relative demands of the writing tasks in 
the two modules (see Table D7). 

As with the reading process component, logs suggest that teachers used a variety of 
strategies to solicit evidence of student understanding and most commonly did so by observing 
and/or collecting and then reviewing student work — be it writing exercises, rough drafts, or other 
assignments. There was also evidence of teachers using peer interactions and feedback as a 
source of information on student understanding. Teachers were most likely to provide feedback 
via one-on-one conferences with students. Other approaches to providing feedback also were 
evident. Overall, teachers seemed to use feedback strategies to a greater extent during the writing 
component of the module than during the reading component (see Tables D8 and D9). 

Teacher Survey Responses 

In this section, we summarize teachers’ responses to the RFA/CRESST implementation 
and scale-up survey. Results are described for a subset of questions that particularly bear on the 
CRESST implementation study, including survey items developed to parallel the log measure 
and those reflecting variables likely to influence classroom implementation. We start with the 
instructional items designed to align with the log measure, and then move to responses related to 
implementation support and sense of efficacy. 

LDC implementation. Survey responses suggest that teachers on average spent relatively 
the most time on their LDC modules’ reading process component, which accounted for about 
40% of their instructional time, followed by the writing process component, which encompassed 
about a quarter of the module instruction. Introducing the module and transition to writing 
components, as would be expected, were allocated relatively less time at 15% and 19% 
respectively. The data, however, show considerable variability by teacher (see Table C6). 

Teachers’ responses regarding their attention to various reading skills and strategies during 
the reading component showed wide coverage. The vast majority of teachers reported giving at 
least some attention to all skill areas queried on the survey, although analysis of rhetorical 
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devices drew considerably less attention than the other skills. The great majority of teachers 
reported giving strong emphasis to citing evidence to support claims, drawing conclusions from 
evidence, and summarizing important points from reading. More than half further reported 
giving a great deal of attention to evaluating the quality of evidence, distinguishing fact from 
opinion, analyzing structure, comparing arguments, examining authors’ perspectives, and note- 
taking. These responses suggest a strong emphasis on critical reading. Independent reading drew 
relatively less emphasis with fewer than half of the teachers reporting placing a great deal of 
emphasis on this activity. This finding stands somewhat in contrast with the log results, which 
suggest that students spent about a third of LDC classroom time reading and conducting research 
independently, and that critical reading drew uneven attention across the two modules (see Table 
Cl). 

Turning to emphasis during the writing component, survey responses also indicate that 
teachers gave substantial attention to a wide range of writing skills — a majority gave at least 
some attention to each of the 10 skills queried by the survey. A sizable majority — more than 
70% — reported giving a great deal of emphasis to supporting their students’ ability to formulate 
a thesis; write introductions, body paragraphs, and conclusions; and incorporate quotes. More 
than half, in addition, reported giving a great deal of attention to formulating counterarguments, 
using transitional words and phrases, and the structure of text (see Table C8). 

Teachers further reported using a variety of strategies for assessing student learning during 
the course of instruction — the vast majority regularly circulated to review students’ work, 
listened to student discussion during group work, asked questions, and collected and reviewed 
students’ written work (see Table C9). 

Teachers’ feedback when they discovered problems in students’ reading and/or writing also 
showed a range of responses. More frequently, teachers reported that they gave students 
additional time to try again and self-correct, with 70% of responding teachers reporting this as a 
frequent response to gaps in understanding. More than half reported often providing students 
specific written comments, providing hints or suggestions, and/or asking a peer to provide 
feedback. Interestingly, survey responses suggest that most teachers tended to provide immediate 
feedback, as only about a third of the teachers reported that they planned to respond to students’ 
difficulties through a future review. Similarly, reteaching (e.g., a repeat of a prior lesson) was not 
a frequently used strategy for most teachers; one fifth of respondents claimed that they rarely or 
never did so. In addition, the majority of respondents rarely or never assigned grammar exercises 
when they noticed problems in student work (see Table CIO). 
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Support for implementation. According to teachers, district administrators demonstrated 
strong support for and understanding of LDC. Teachers’ perceptions of their school 
administrators’ support for the intervention, however, varied widely. Some teachers reported that 
their administrators understood LDC and how it aligned with school priorities, communicated its 
importance, and provided feedback on instruction, and others reported a lack of knowledge, 
support, and feedback. Majorities of teachers reported having their classrooms visited by their 
principal, instructional coach or department head, and teacher colleagues during LDC module 
instruction, and 40% reported receiving a visit from a district LDC project leader (see Tables 
Cll and C 12). 

Over three quarters of teachers reported participating in formal professional development 
(PD) related to LDC in the 2012-2013 school year. This group of teachers participated on 
average in about two sessions, with a range between one and four sessions. Professional 
development was most commonly delivered in districtwide or small group meeting settings. 
Teachers generally found all settings to be effective. Teachers reported that LDC professional 
development covered a wide variety of topic areas. The most commonly cited professional 
development foci were using LDC as a way to implement Common Core, building a teaching 
task, using the instructional ladder, using mini-tasks to address skills in reading and writing, 
providing feedback on writing, and using rubrics to score student work. Teachers did not 
commonly report that they received PD on implementing LDC with high-needs students, for 
example, English language learners, special education students, or students reading or writing 
below grade level (see Tables C13 through Cl 6). 

Teachers also reported spending a considerable amount of time collaborating with teacher 
colleagues on LDC. Almost two thirds of respondents reported having regularly scheduled 
common planning time with colleagues to discuss LDC. Furthermore, a majority of teachers 
participated in both scheduled meetings and informal discussions around LDC at least every 
other week. Although there was variation across teachers, on average teachers had positive 
attitudes regarding this collaboration. Teachers generally felt that their LDC colleagues were 
collaborative. In addition, most teachers found this collaboration at least somewhat helpful in 
effectively using the LDC framework, better supporting student learning, and teaching LDC 
modules (see Tables C17 through C 19). 

Perhaps as a result of available professional development and collaborative opportunities, 
teachers felt confident in their capacity to implement the LDC modules. Most reported knowing 
which skills students needed to complete the task, and the types of mini-tasks that would help 
students prepare for the task. Some teachers, however, did report barriers to successful 
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implementation of LDC modules. Many teachers felt they did not have sufficient time to prepare 
for teaching the modules and to respond to student writing (see Tables C20 and C21). 

Attitudes toward LDC efficacy. Overall, most district teachers seemed to find LDC to be 
a helpful and effective teaching tool. Large majorities of teachers reported that LDC helped them 
meet a wide range of instructional goals, including implementing the CCSS, increasing the rigor 
of writing assessments, engaging students, practicing formative assessment, and teaching literacy 
and subject matter content. Similarly, although there was variation in responses, teachers 
typically reported that the LDC framework was effective in promoting literacy instruction in 
secondary and content area classrooms, improving literacy, addressing the Common Core, and 
encouraging the use of formative assessment (see Tables C22 and C23). 

Teachers were varied in their opinions regarding their students’ level of engagement with 
LDC. Just over half of teachers reported that students were more engaged during LDC module 
instruction than in regular instruction, while a third of teachers felt their level of engagement was 
the same, and about 15% reported students were less engaged in LDC than other instruction. 
Nearly all teachers reported that their students experienced at least some success on reading 
mini-tasks, writing mini-tasks, and the final writing tasks; however, considerably more teachers 
reported that their students experienced a great deal of success on the final writing tasks than on 
the mini-tasks making up the instructional ladder. Based on these responses, teachers appear 
highly satisfied with the effectiveness of the LDC modules, but teachers and their students may 
need help to increase the productivity of the reading and writing mini-tasks (see Tables C24 and 
C25). 

Finally, teachers in the district gave a strong endorsement regarding the impact of LDC on 
student learning. Although there was variation, most teachers agreed that LDC had resulted in 
higher quality student writing and had supported students’ college readiness. When reflecting on 
their most recent module, over 90% of teachers reported that a majority of students improved 
their content knowledge, and over 85% of teachers reported a majority of students improved 
their literacy skills (see Tables C26 and C27). 

Module Ratings 

Here we present findings from the scoring of the four primary modules used in sixth-grade 
Advanced Reading, as displayed in Table 9. On average, the analysis found the modules to be of 
high quality; module quality dimensions were rated at least moderately present or realized (a 
score of 3) and more often approaching or at sufficiently present or realized (a score of 4). The 
results, however, show substantial variation in quality across modules, with Module 1, Personal 
Change, showing a lower average rating and the lowest ratings on each individual dimension. 
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Reviewers found the content relatively less aligned with the Common Core’s major shifts and 
less rigorous than the other modules on a number of dimensions. This lesser rigor may have been 
an explicit strategy to start with relatively easier content and then introduce students more 
gradually to Common Core expectations over the series of modules. Module 2, Savage Harvest, 
and Module 3, Communication in a Cybernation, on the other hand received strong overall 
ratings. 

The data also reveal considerable variation in quality across the nine dimensions. For 
example, module coherence, fidelity to LDC, effectiveness of writing task, and text alignment 
received high ratings overall. Alignment to literacy standards ratings were more variable; 
according to raters, the modules consistently included specific standards to which the modules 
were to be aligned, but alignment ratings suffered when raters could not identify strong evidence 
of those standards in the activities in which students engaged nor in the work students produced. 
Ratings for text appropriateness and text rigor were somewhat lower than for other dimensions. 

In reviewing these findings, it is important to note that the modules have been subsequently 
revised. Moreover, module ratings here are significantly higher than those in the companion 
study of LDC implementation in eighth- grade history/social studies and science courses. The 
latter finding may suggest the value for module quality of a centralized approach to module 
development that involves literacy specialists. 
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Table 9 


District Advanced Reading Module Scores 


Dimension 

Module 1: Personal 
Change Creates 
Community 

Module 2: 
Savage Harvest 

Module 3 : 
Communication 
in a Cybernation 

Module 4: 
Fit for Life 

Average 

module 

ratings 

Effective writing task 

3.0 

4.0 

5.0 

3.0 

3.8 

Alignment to literacy 
standards 

2.5 

3.0 

3.0 

4.0 

3.1 

Text alignment 

3.0 

4.5 

4.0 

4.0 

3.9 

Text appropriateness 

2.5 

4.0 

4.0 

3.0 

3.4 

Text rigor 

2.5 

4.0 

4.0 

2.5 

3.3 

Fidelity to LDC module 
instruction 

4.0 

4.0 

4.0 

4.0 

4.0 

Quality instructional 
strategies 

3.0 

5.0 

4.0 

3.0 

3.8 

Coherence and clarity of 
module 

3.0 

5.0 

3.0 

4.0 

3.8 

Overall impression 

3.0 

4.0 

3.0 

3.0 

3.3 

Average score across 
dimensions 

2.9 

4.2 

3.8 

3.4 

3.6 


Note. Ratings are on 1-5 point scale, where a score of 1 indicates that a dimension is not in evidence, a score of 3 
indicates that quality was moderately realized, and 5 indicates that quality is fully realized. 


Summary of Implementation Data 

Teachers’ responses to surveys and logs provide a portrait of by whom, how, and with what 
support LDC was implemented by the study sample, as well as participating teachers’ 
impressions of effectiveness. 

Who. Survey results indicate that study teachers were highly diverse in their prior 
experience, ranging from novice first- or second-year teachers to 40-year veterans. Almost all 
had experience in teaching diverse learners — from struggling to advanced literacy learners, 
students with disabilities, and English language learners. 

Most of the study teachers had one to two years of experience implementing four to five 
LDC modules each of these years. Although teachers were required to participate in LDC, rather 
than having volunteered to do so, they appeared enthusiastic about their participation (see below 
regarding attitudes). 

How. Teacher log, survey, and module review data indicate that teachers followed the LDC 
framework. As indicated by the logs and surveys, although independent reading and/or writing 
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were the most frequently occurring activities during LDC instruction, students engaged in a wide 
variety of reading and writing skills during this time. The log data indicate that direct strategy 
instruction and attention to critical reading — such as evaluating evidence, comparing arguments, 
and examining author’s purpose — may need additional attention. However, variation in skills 
addressed from one module to the next and teacher survey responses suggest a more 
differentiated emphasis across modules. That is, while the analysis of one module may show 
little attention to a topic or skill, that topic or skill may be addressed in a subsequent module; for 
example, log data suggest that teachers’ emphasis moved from general to more specific aspects 
of writing from Savage Harvest to Cybernation. Across all modules, our review showed attention 
to a wide range of reading and writing skills, consistent with teacher survey responses. The 
coordination of literacy skills across modules raises an important point in LDC implementation: 
That is, addressing the full range of ELA standards requires careful engineering and coordination 
across modules. 

Across both reading and writing, teachers reported engaging in frequent formative 
assessment, involving multiple strategies for monitoring student learning and for responding to 
student misunderstandings as they occur. It is noteworthy that the modules specified a variety of 
specific assessment activities (and success criteria) and assessment techniques for each lesson, 
and many included exit slips, to support teachers’ formative assessment practices. 

Analysis of the district’s modules provides another window into the quality with which 
LDC is being implemented. The modules generally received strong ratings for fidelity to the 
LDC framework, module coherence, effectiveness of writing task and text alignment. Ratings for 
all the nine dimensions examined approached or achieved high quality. Results, however, also 
indicated variation across modules and dimensions of quality. 

With what support. Survey responses indicated that teachers felt their district leadership 
strongly supported the LDC intervention, but that the support of school administrators was more 
variable across the sample. Leadership support is evident in the LDC professional development 
that the majority of teachers reported participating in during the study year and in the regular 
scheduling of joint planning time for LDC teachers. In fact, teachers reported meeting formally 
or informally with their peers about LDC at least every other week, and most found that these 
various collaborative activities were helpful in implementing LDC. Although teachers felt 
capable of implementing the modules and knowledgeable about the skills and mini-tasks that 
were required, many teachers felt that they did not have sufficient time to prepare for teaching 
the modules and to respond to students’ writing. 
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Attitudes toward LDC. Teachers reported that they found LDC a helpful and effective 
tool in meeting a variety of goals, including implementing the Common Core State Standards, 
using formative assessment, incorporating literacy into content classrooms, and increasing the 
rigor of their writing assignments. Most teachers also felt that LDC resulted in higher quality 
student writing and supported students’ college readiness. When reflecting on their most recent 
module, over 90% of teachers reported that a majority of their students improved their content 
knowledge, and over 85% of teachers reported a majority of students improved their literacy 
skills. 
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Chapter 4: Student Learning Results 

The study used multiple measures of student learning to examine LDC effects and to 
explore relationships between LDC implementation variables and student outcomes. We first 
present descriptive results for the CRESST Integrated Learning Assessment (ILA) and state and 
district assessment for the LDC sample only. We then present the results of the quasi- 
experimental matched control group analysis of LDC effects on student learning, followed by the 
within-district regression discontinuity analysis. 

Descriptive Results 

Descriptive results of Child Labor Integrated Learning Assessment. As noted earlier, 
study teachers who completed logs were asked to administer an ILA specifically developed to 
align with content in the Savage Harvest module, the second of the four required modules, which 
focuses on contemporary international child labor. The first component of the ILA calls for 
students to read and respond to a variety of texts through multiple choice and short extended 
response questions; the second component poses a prompt in which students are asked to 
combine their existing knowledge of the topic and new information garnered from the ILA 
readings to create an extended expository essay. These essays are scored on five dimensions: 
content understanding, rhetorical structure/quality, organization, reference support with text, and 
grammar and conventions, with each rated on a four-point scale. Scores on the five dimensions 
were summed to arrive at an overall score for the essay (see Chapter 2 for additional detail). 

Overall, as shown in Table 10, student performance on both the reading and writing 
components of the Child Labor ILA was highly variable. On the reading component, students 
ranged from answering only one question correctly to achieving perfect scores. Scores on the 
writing component scores ranged from five to 17 out of 20 possible points. Student scores 
averaged 1 1 points for reading, or just over two thirds of the total and 1 1.5 of 20 total possible 
points in writing, equal to approximately 57% of the total possible score. 

Table 10 


Descriptive Results of Child Labor Integrated Learning Assessment 


Component 

Number of students 

Total possible score 

Mean score 

SD 

Minimum 

Maximum 

Reading 

974 

16 

11.00 

2.30 

1 

16 

Writing 

938 

20 

11.49 

2.89 

5 

17 
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Data in Table 11 suggest that students performed at similar levels across all five writing 
dimensions. Although there was considerable variation across students, on average students 
performed between the basic and proficient levels, based on rubric criteria. Interestingly, scores 
were highest on average for content understanding, even though no student received the highest 
possible score (4) on this dimension. Mean scores across the four writing dimensions were in the 
basic range. 


Table 1 1 


Descriptive Statistics for Child Labor Writing Task Score Dimensions (n = 938) 


Dimension 

Mean 

SD 

Minimum 

Maximum 

Content understanding 

2.52 

0.628 

1 

3 

Rhetorical structure/quality 

2.13 

0.803 

1 

4 

Organization 

2.22 

0.787 

1 

4 

Reference/support with text 

2.24 

0.709 

1 

4 

Grammar and conventions 

2.35 

0.635 

0 

4 


Florida Comprehensive Achievement Test (FCAT 2.0). As described earlier, the study 
drew on students’ end-of-year perfonnance on the reading FCAT. Table 12 provides descriptive 
data on student performance on the FCAT at the end of the study year, 2012-2013, and the prior 
year. The sample of 6926 students includes all individual students taking Advanced Reading 
classes in the study district with available FCAT scores for both years. The results suggest that 
students whose teachers participated in LDC scored considerably higher than the statewide 
average in both 2012-2013 (state mean FCAT Reading = 225) and 2011-2012 (state mean 
FCAT Reading = 221). This is not surprising given that LDC was implemented in Advanced 
Reading classes and therefore students at the lower end of the achievement distribution in the 
study district would not be included in their mean. 
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Table 12 


Sixth-Grade Advanced Reading Students ’ Performance on FCAT Reading and District Writing Scores 


Variable 

Number of students 

Mean 

SD 

Minimum 

Maximum 

FCAT reading 

Spring 2013 

6926 

236.5 

16.73 

167 

283 

Spring 2012 

6926 

233.7 

14.77 

187 

277 

District writing 

Spring 2013 

6619 

2.2 

0.91 

0 

4 

Fall 2012 

6577 

1.69 

0.73 

0 

4 


Local district writing measure. The district writing results, seen in Table 12, indicate that 
sixth-grade Advanced Reading students started the year at a rudimentary level of writing. By the 
end of the year their writing showed significant improvement. However, a score of 2 indicates 
that the writing has weak focus and organization, lacks adequate support, and is limited in 
language. 

Correlation between outcome measures. Before moving to our analysis of the effects of 
LDC on student learning, we report on the correlations between the four study measures 
addressing student learning outcomes: ILA writing, ILA reading, FCAT prior year and outcome 
year scores, and district writing pre- and post-test scores (see Table 13). In viewing these results, 
it is important to note that both the ILA and the district writing assessment are based on a single 
task, which limits the generalizability of the scores and thus the relationship to other measures. 
And indeed the district writing assessment shows the lowest correlation with other measures. The 
pattern of relationships suggests the distinctness of the constructs measured by each instrument: 
That is, the ILA is a writing-from-reading measure, while the FCAT focuses on reading 
comprehension. It is of interest that the spring FCAT scores show a stronger relationship with 
ILA scores, which are from the same time period, than do the FCAT results from the prior year. 
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Table 13 


Correlation Between ILA, FCAT, and District Writing Scores 



ILA 

ILA 

ILA 

FCAT, 

FCAT, 

District writing, 

District writing, 


reading 

writing 

total 

spring 2012 

spring 2013 

spring 2013 

fall 2012 

ILA reading 

— 







ILA writing 

0.39** 

(934) 

— 






ILA total 

0.78** 

0.88** 

— 






(934) 

(934) 






FCAT, spring 

0.39** 

0.38** 

0.45** 

— 




2012 

(872) 

(841) 

(839) 





FCAT, spring 

0.48** 

0.47** 

0.55** 

0.63** 

— 



2013 

(872) 

(841) 

(839) 

(877) 




District writing, 

0.23** 

0.37** 

0.37** 

0.34** 

0.37** 

— 


spring 2013 

(812) 

(782) 

(780) 

(817) 

(817) 



District writing, 

0.09* 

0.16** 

0.16** 

0.19** 

0.21** 

0.13** 

— 

fall 2012 

(822) 

(793) 

(791) 

(827) 

(827) 

(785) 



Note, n presented in parentheses. 

*p < .05, two-tailed. **p < .01 level, two-tailed. 


Quasi-Experimental Analysis Overview 

Two different types of quasi-experimental designs were utilized in the study to estimate the 
effect of LDC on student achievement: a matched control group design and a regression 
discontinuity design. The matched control group analyses compare the full population of LDC 
students in sixth-grade Advanced Reading courses in the study district to selected matched 
students from across Florida in Advanced Reading or language arts classes. Coarsened Exact 
Matching (CEM), described later in this chapter, was used to select matched students with 
similar demographic and prior achievement characteristics in schools with similar prior 
effectiveness. The regression discontinuity design takes advantage of a natural experiment 
created by the study district’s selection process for entry into the Advanced Reading course. The 
design focuses on students near the cut-point for entry into Advanced Reading, and compares the 
performance of students that just made it into the course to students who just missed being 
assigned to the course. 

Each of the two designs has advantages and disadvantages. The matched control group 
design includes a much more complete set of students receiving LDC instruction in sixth-grade 
Advanced Reading. However, because LDC was implemented districtwide in the study district 
and matched control students were selected from outside the district, it is difficult to tease out the 
impact of LDC from the effect of other district programs and conditions. That is, we cannot rule 
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out the possibility that differences between treated and matched control samples are due to other 
district-specific effects. With the regression discontinuity design, these district effects are 
controlled by nature of the fact that both treatment and control group students come from the 
district. However, the sample of students comes from a specific portion of the prior achievement 
distribution directly around the prior score cut-point, and therefore the estimates may not be 
generalizable to the full population of LDC students. Another advantage of the regression 
discontinuity design is that it allows us to test the effect of LDC on the local district writing 
measure, which is not possible in the matched control group analysis (because the district- 
specific writing measure is not available for students outside the district). 

As described earlier, schools in the study district began implementing LDC in two stages. 
Ten Phase 1 pilot schools started implementing LDC in 2010-2011, with the remaining middle 
schools in the district beginning implementation in 2011-2012. The different quasi-experimental 
analyses tested the impact of LDC on both cohorts of schools. The primary results presented in 
this chapter, however, focus on the impact of Phase 2 schools in 2012-2013. In the case of the 
matched control group analysis, it was necessary to exclude Phase 1 schools from the 2012-2013 
analysis to properly control for the effectiveness of schools prior to LDC intervention. 

Matched Control Group Design 

Modeling summary. Two different types of hierarchical linear models (HLM) were 
employed in our matched control group analyses: a three-level model with student at Level 1, 
school by year at Level 2, and school at Level 3, and a two-level model with student at Level 1 
and school at Level 2. The three-level model used 2009-2010 as the baseline year and estimated 
effects in each subsequent year — 2010-2011, 2011-2012, and 2012-2013 — as compared to 
baseline. This model provided information on achievement trends for study district Advanced 
Reading students relative to matched controls across all of the years of implementation. The 
analysis showed a large dip in perfonnance in the study district from 2009-2010 to 2010-2011 
relative to statewide control students for students in both cohorts of schools: the Phase 1 pilot 
schools that began implementing LDC in that year and Phase 2 schools that hadn’t begun LDC 
implementation yet. Given that this negative effect was found for both Phase 1 and Phase 2 
students, it is unlikely to have been the result of LDC implementation, but rather the result of 
other unexplained district factors or conditions. 

As mentioned earlier, one possible explanation for the negative effect in 2010-2011 is the 
introduction of the new state reading test, FCAT 2.0. Decreases in performance are common 
with the introduction of new assessments, and the study district may have been impacted by this 
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shift in a differential way from the state at large due to unexplained district conditions. See Table 
El in Appendix E for the 2012-2013 effect estimates from the three-level model. 

Given the overall drop in performance in the study district in 2010-2011, and the fact that 
the state assessment changed in the same year, our main analysis used data only from the new 
test (starting with 2010-2011). We implemented this with a two-level HLM model estimating 
LDC impact in 2012-2013, while controlling for student prior achievement in 2011-2012 and 
the prior effectiveness of schools in 2010-2011. We exclude the 10 pilot schools from this 
analysis, as they had already started implementing LDC in the baseline year. 

Methodology for two-level HLM analysis. Treatment students were not randomly 
selected to participate in the LDC initiative. To estimate the impact of LDC it is therefore 
necessary to control for the effects of student and school characteristics. One way to control for 
these characteristics is to use matching techniques to identify a group of comparison students 
who are demo graphic ally and academically similar to the intervention students. Our matching is 
conducted at the student level and accounts not only for student demographics and prior 
achievement, but the prior effectiveness of schools as well. 

We employ a matching technique known as Coarsened Exact Matching (CEM) to identify 
comparison students. Coarsened Exact Matching is a flexible matching approach with many 
favorable properties, and allows the researcher to specify precise conditions under which a 
comparison student may be matched with an intervention student. For categorical variables such 
as race/ethnicity or free/reduced price lunch status, this often entails exact matching, while for 
continuous measures, such as prior outcomes, cut-points for matching can be specified. With this 
approach we can set precise cut-points on the most important indicators such as prior academic 
achievement to ensure that where possible every treatment student is matched with a suitable 
comparison. 

Table 14 summarizes the variables used for the matching. Although we match on both 
student and school characteristics, all matching is at the student level. In addition to student 
demographics and prior achievement, we also match on the prior effectiveness of the student’s 
school, as measured by the school’s value added to student learning in 2010-2011, calculated 
using standard methods. That is, students whose teachers participated in LDC were matched to 
students in other districts with similar demographics and prior achievement, and who were 
attending similarly effective schools. A school’s value added in 2010-2011 was obtained by 
running a two-level HLM model that controlled for student characteristics such as prior 
achievement, gender, and free and reduced price lunch eligibility at Level 1, and saving out 
empirical Bayes estimates of school value added at Level 2. 
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Table 14 


Summary’ of Matching Variables 


Indicator type 

Variable 

Student 

Gender 

Student 

White 

Student 

Hispanic 

Student 

Black 

Student 

Asian 

Student 

Free/reduced price lunch 

Student 

English language speaker 

Student 

Spanish language speaker 

Student 

Prior achievement in reading 

School 

School prior effectiveness 


The CEM process was successful in finding matches for a large majority of the eligible 
5,548 LDC students, as seen in Table 15. The matching models were effective both in retaining a 
large percentage of treatment observations (96.2), and in achieving close balance with regard to 
prior student scores and demographics, as well as the school effectiveness indicator (see Table 
E2 in Appendix E for prior achievement and demographic characteristics of eligible and matched 
treatment and comparison samples). 


Table 15 


Summary’ of Treatment and Comparison Samples 


Sample 

Treatment 

Comparison 

Eligible for matching 

5,548 

14,523 

Matched sample 

5,338 

9,241 


Student demographic and prior achievement variables, as well as school prior 
effectiveness, are included as controls in the HLM regression model as well as the matching 
protocol. Our estimates therefore control for observables in two ways, at the matching and 
modeling stages. The model also included interactions between the LDC treatment indicator and 
student characteristics to test whether LDC had differential effects on student learning depending 
on the student’s or school’s characteristics. The interaction analyses should be considered 
exploratory and results treated as tentative, given sample size and other data limitations. 
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HLM results. Results for the two-level HLM presented in Table 16 show no evidence of 
an impact of LDC on FCAT reading perfonnance. The coefficient for the treatment effect is 
negative but statistically insignificant. The interaction of treatment status with prior student 
achievement is positive and statistically significant at the 5% level, suggesting that LDC may be 
more effective for students with higher levels of prior achievement. This is consistent with 
findings in our companion study of LDC in eighth-grade history/social studies and science 
classes (Hennan et al., 2015). We did not find statistically significant interaction effects for other 
demographic variables. Note that only the main variables of interest are reported in Table 16. 
The complete results can be found in Table E3 in Appendix E. The main specification reported 
here excludes schools without prior effectiveness data in 2010-2011 (i.e., new schools). An 
alternative specification including these schools found similar results and is reported in Table E4. 

Table 16 


2012-2013 LDC Student Effect Estimates on FCAT Reading, Including Interactions 
With Student Characteristics 


Level 2 variables 

Model coefficient (SE) 

LDC treatment 

LDC treatment by student characteristics interactions 

-0.051 (0.048) 

Gender 

0.073 (0.038) 

Free/reduced price lunch eligible 

-0.049 (0.038) 

Prior achievement 

0.056 (0.025)* 


Note. Fixed effects for demographic predictors and for prior school effectiveness 
not shown. 

*p=. 05. 


Regression Discontinuity Design 

As described earlier, a regression discontinuity design (RDD) allowed us to examine the 
impact of LDC implementation on student achievement within the study district by exploiting a 
natural experiment created by the use of a cut-point to assign students into Advanced Reading 
courses (and therefore LDC). The regression discontinuity analyses tested the effect on both 
FCAT reading scores and local district writing scores. Students were assigned to Advanced 
Reading primarily based on prior year FCAT reading scores. Students above a certain threshold 
were significantly more likely to be assigned to Advanced Reading, compared to students just 
below this threshold. The RDD approach exploits this large change in the probability of being 
assigned to Advanced Reading as students cross the threshold to estimate the impact being 
assigned to Advanced Reading. If LDC had effects on student learning, we might expect to see 
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the benefit of being assigned to Advanced Reading increase after LDC implementation. We 
implemented this analysis for four school years: 2009-2010, 2010-2011, 2011-2012, and 2012- 
2013. As noted earlier in the report, the LDC intervention began with 10 Phase 1 pilot schools in 
2010-2011, followed by expansion to all middle schools in the district in 2011-2012 and 2012- 
2013. For the 2010-2011 analysis, we removed students in the 10 Phase 1 pilot schools in 2010- 
2011. This methodological decision allowed us to analyze two cohorts of students prior to LDC 
intervention and two cohorts of students in the post-intervention period. 

We implemented several standard tests to detennine whether our setting was amenable to 
the RDD. We conducted tests to determine 

• if the assignment measure (FCAT pre-score) was used to determine assignment to 
Advanced Reading, with an apparent threshold at a cut score, 

• what the threshold on the assignment measure was, 

• whether the density of the assignment measure was continuous through the threshold 
indicating potential manipulation/gaming of the assignment measure, and 

• whether the observable characteristics of students were continuous through the 
threshold. 

Having taken the proper pre-design steps we then proceeded with RDD analyses. Analytic 
steps included identifying a bandwidth of students around the assignment cut-point for inclusion 
in each model and producing a local average treatment effect estimate and standard error. In 
addition, a graph plotting the assignment measure and outcome estimate was used for visual 
inspection of potential discontinuity. 

Diagnostics on assignment measure. The primary measure used for determination of 
assignment in Advanced Reading is the prior year FCAT reading score. From graphical and 
frequency-based analyses it is clear that while there is no single cut-point that correctly classifies 
every student into Advanced Reading classes, there are cut-points that fairly sharply assign the 
students. For example, in 2012-2013 only 2.1% of students with a prior 2011-2012 FCAT score 
of 214 or 215 were assigned into Advanced Reading, in contrast to 96.0% of students with a 
prior 2011-2012 scale score of 216 or 217. Thus we use a cut-point of 216 for the 2012-2013 
RDD analysis. For the RDD analyses in 2009-2010, 2010-2011, and 2011-2012, the FCAT 
prior score was scaled differently, and a scale score of 1515 assigned students into Advanced 
Reading nearly as sharply as the score of 216 did in 2012-2013. Graphical analysis also verified 
the discontinuity in the probability of Advanced Reading assignment on either side of the cut- 
point suggesting that a “fuzzy” RDD would be appropriate. Figure 1, graphing the relationship 
between FCAT prior year scores and assignment into Advanced Reading in 2012-2013, clearly 
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demonstrates this discontinuity. See Figures El, E2, and E3 in Appendix E for similar graphs for 
other analysis years. 

To assess discontinuity in the density of observations at the cut-point, we performed a 
McCrary Test and visually examined density graphs in each year. Neither the analytic test nor 
visual inspection provided any evidence of discontinuity or manipulation around the cut-points. 
Overall the diagnostics suggested we had appropriate data to conduct RDD analyses. 



- 2-10 1 2 

Standardized Fifth Grade FCAT Reading, 2011-12 


Figure 1. Relationship between assignment variable (fifth-grade reading in 2011-2012) and probability of 
assignment into Advanced Reading in 2012-2013. 

Results for FCAT reading development. Regression discontinuity results for each of the 
four years are presented in Table 17. The assignment variable was standardized around a cut- 
point score of zero to ease comparison across years, as the scale of the FCAT score changed over 
time. The bandwidths shown in Table 17, which detennine the observations included in the 
analysis, were obtained using the Imbens and Kalyanaraman optimal bandwidth algorithm. The 
results indicate no evidence of a discontinuity in outcomes around the threshold for assignment 
to Advanced Reading either in the years before or the years after the LDC intervention began. 
Figure 2 shows this graphically for 2012-2013, the key post-intervention year. There is no 
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Standardized Sixth Grade FCAT Reading, 2012-13 


evidence of discontinuity at the standardized cut-point of zero on the prior FCAT reading score 
(the assignment variable). Graphs for other years can be found in Appendix E (see Figures E4, 
E5, and E6). 


Table 17 


LDC Student Regression Discontinuity Design Local Average 
Treatment Effect Estimates on Sixth-Grade FCAT Reading Scores, 
2009-2010 to 2012-2013 


Year 

Bandwidth 

Observations 

Estimate (SE) 

2009-2010 

1.032 

8,324 

0.031 (0.035) 

2010-2011 

0.749 

5,136 

-0.025 (0.057) 

2011-2012 

0.692 

6,112 

-0.015 (0.029) 

2012-2013 

0.597 

6,313 

-0.002 (0.022) 



Standardized Fifth Grade FCAT Reading, 2011-12 

Figure 2. Regression discontinuity graph for 2012-2013 showing relationship between assignment variable (fifth- 
grade FCAT reading in 2011-2012) and the outcome variable (sixth-grade FCAT reading in 2012-2013). 
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Results for FCAT writing development. Regression discontinuity results for the share of 
students achieving a basic level of proficiency on the district writing assessment are presented in 
Table 18. As described in Chapter 2, the district writing assessments were scored using a six- 
point rubric in 2010-2011 and 2011-2012 based on the rubric for the state writing test. In 2012- 
2013, the study district began using the LDC four-point rubric to score writing assessments. 
Basic perfonnance constituted a score of 3 on the earlier rubric and a score of 2 on the later 
rubric. Otherwise the methodology is similar to that used for the analysis of FCAT reading 
outcomes. As with reading, we see no evidence of a discontinuity in the share of students who 
have at least a basic level of proficiency according to the writing assessment at the threshold for 
Advanced Reading assignment either in the year before or the years after the LDC intervention 
began. Figure 3 shows this graphically for the primary outcome year of interest, 2012-2013. 
Graphs for other years can be found in Appendix E (see Figures E7 and E8). 

Table 18 


LDC Student Regression Discontinuity Design Local Average 
Treatment Effect Estimates on Probability of Scoring at a Basic 
Level on the District Writing Measure 


Year 

Bandwidth 

Observations 

Estimate (SE) 

2010-2011 

0.968 

5,378 

0.043 (0.032) 

2011-2012 

1.006 

6,986 

0.001 (0.026) 

2012-2013 

0.574 

5,299 

0.028 (0.034) 
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Chapter 5: Implementation Variables Related to LDC Success 


In this chapter, we report on analyses of the relationship between variables derived from 
our implementation measures and student outcomes. Drawing on data from the teacher logs and 
surveys, we explored a variety of composite implementation variables and examined their 
relationship to student learning outcomes. When more standard regression analyses failed to 
produce stable patterns of results, we investigated the extent to which a variety of 
implementation variables differentiated LDC teachers at relatively high, middle, and low levels 
of effect on student learning, based on teachers’ estimated value added during the study year. We 
then compared the mean scores on each variable using an ANOVA difference in means test. 
These processes are further described in the following section. Because these analyses are 
exploratory, particularly given the small sample sizes, results should be interpreted with caution. 

Identification of Implementation Variables 

Both substantive theory and psychometric analysis guided the development of composite 
variables. Our identification of priority variables centered on evidence-based teacher practices 
that were likely to influence student learning (e.g., Heritage, 2010; Herman, Osmundson, Dai, 
Ringstaff, & Timms, 2011; Hinchman & Sheridan-Thomas, 2008) and on variables that 
influence the implementation of new practices and programs — for example, teacher beliefs, 
sense of efficacy, leadership support, collaboration, and professional development (see for 
example, Fullan, Hargreaves, & Lieberman, 2010; O’Day, Bitter, & Gomez, 2011; Supovitz & 
Weinbaum, 2008). Through cycles of hypothesis generation and a variety of exploratory factor 
analyses (EFA) and cluster analyses, we identified 18 variables for additional study. 

These variables, their sources, and operational definitions are shown in Table 19. 
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Table 19 


Teacher-Level Implementation Variables Used in Within-Treatment Analyses 


Instrument 

source 

Variable 

Description 

Teacher log 

Range and intensity of 
reading instruction 

Sum of reading skills reported for each teacher log in which reading was 
addressed. A mean total score for each teacher was then computed as the 
average across all relevant logs. Scores for each reading skill indicated 
the emphasis it was given that day: focus of student work = 2; touched on 
briefly = 1 ; not today = 0. 

Teacher log 

Attention to close 
reading of text 

Mean sum of reading skills reported representing high-level analysis of 
text on logs for which reading was addressed. Scores for each close 
reading item indicated the emphasis it was given that day \ focus of 
student work = 2; touched on briefly = 1; not today = 0. 

Teacher log 

Attention to basic 
reading skills 

Mean sum of basic reading skills items reported on logs for which 
reading was addressed. Scores for each basic reading skill item indicated 
the emphasis it was given that day: focus of student work = 2; touched on 
briefly = 1 ; not today = 0. 

Teacher log 

Attention to writing 
skills 

Mean sum of all writing skills reported on logs for which writing was 
addressed. Coding: focus of student work = 2; touched on briefly = 1; not 
today = 0. 

Teacher log 

Range and intensity of 
formative assessment 
of student learning 

Mean sum of all formative assessment practices reported in logs 
addressing reading and/or writing. Scores for each formative assessment 
practice indicated the extent to which it was used: to a great extent = 2; 
to some extent = 1; not at all = 0. 

Teacher log 

Range and intensity of 
feedback to students 

Mean sum of all practices for providing feedback to students based on 
student work in reading and writing. Scores for each feedback practice 
indicated the extent to which it was used: to a great extent = 2; to some 
extent = 1 ; not at all = 0. 

Teacher log 

Range and intensity of 
teacher literacy 
practices (reading 
skills, writing skills, 
formative assessment) 

Continuous variable measuring the extent to which teachers reported 
attention to reading skills, writing skills, formative assessment practice, 
and providing feedback. Each of these four domains was weighted 
equally to create the variable. 

Teacher log 

Teacher log cluster 
variable: high quantity 
literacy practice 

Cluster binary variable distinguishing teachers who reported conducting 
a greater quantity of practices during LDC module instruction from 
teachers who reported a smaller quantity of practices. Variable was 
created by first performing cluster analysis on individual items in each 
domain (reading skills, writing skills, formative assessment), and then 
performing a second cluster analysis using the identified cluster 
variables. 

Teacher survey 

Factor 1: Attention to 
close reading of text 

Factor 1 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Factor 1 reflected reading items related to close 
reading of text. Variable confirmed and tested for reliability using 
confirmatory factor analysis. 
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Instrument 

source 

Variable 

Description 

Teacher survey 

Factor 2: Attention to 

paragraph 

writing/structure 

Factor 2 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Variable reflected writing items related to 
paragraph construction and structure of writing. Confirmed and tested for 
reliability using confirmatory factor analysis. 

Teacher survey 

Factor 3: Teacher-led 
formative assessment 
practice 

Factor 3 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Variable reflected teacher-oriented formative 
assessment practices. Confirmed and tested for reliability using 
confirmatory factor analysis. 

Teacher survey 

Factor 4: Peer-oriented 
formative assessment 
practice 

Factor 4 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Variable reflected student-to-student formative 
assessment practices. Confirmed and tested for reliability using 
confirmatory factor analysis. 

Teacher survey 

Total modules taught in 
201 1-2012 and 2012- 
2013 school years 

Sum of responses to Questions 16 and 17 in teacher survey. 

Teacher survey 

Support for teaching 
literacy in content area 
classrooms 

Mean across three items addressing content teachers’ time and 
responsibility for teaching literacy. Coding: disagree = 0; disagree 
somewhat = 1; agree somewhat = 2; agree = 3. 

Teacher survey 

Teachers’ perceived 
capacity to teach LDC 

Mean response to questions about teacher efficacy (Question 26) and 
barriers with regard to LDC (Question 39). Coding: disagree = 0; 
disagree somewhat = 1; agree somewhat = 2; agree = 3, with Items 39c- 
g reverse coded. 

Teacher survey 

District and school 
support for LDC 

Mean response to items about various ways that district and school 
leadership show support for LDC (Question 43). Coding: disagree = 0; 
disagree somewhat = 1; agree somewhat = 2; agree = 3. 

Teacher survey 

Utility of teacher 
collaboration 

Mean response to items asking about extent and helpfulness of teacher 
collaboration in implementing LDC (Question 49). Coding: disagree = 0; 
disagree somewhat = 1; agree somewhat = 2; agree = 3. 

Teacher survey 

Professional 
development dosage 

The number of formal scheduled LDC PD sessions in 2012-2013 
(Question 55). 


As the table shows, the implementation analyses included a number of teacher-level 
implementation variables created from teacher log responses. The variables are summary 
measures of teacher responses in four key domains of the log, which also represent component 
emphases for LDC: Teachers’ daily focus on reading skills, teachers’ daily focus on writing 
skills, teachers’ daily use of strategies to assess student learning, and teachers’ daily use of 
strategies to provide feedback to students. The latter two domains together constitute our 
measure of formative assessment practice. Mean sum variables were created for each domain, 
based on both the number of skills or strategies the teacher reported when reading and/or writing 
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was addressed and the depth of attention reportedly given to the skill (e.g., on the writing 
variable, a writing skill would be coded as 2 if the teacher reported it was a primary focus on the 
day of the log, and a 1 if the teacher reported that it was only touched on briefly). For the reading 
skills domain, we also separated mean sum variables for two subgroups of items: those 
emphasizing close reading of text and those addressing more basic reading skills. The decision to 
analyze this domain at a finer level of detail was based on both our theoretical assumptions 
regarding the relative importance of skill development in these two areas and exploratory 
analysis of the log and survey data that provided evidence of the dichotomy. 

Finally, we included two variables that attempt to capture variety in the teachers’ reported 
attention to all four domains. One variable is the mean sum of activity reported across all four 
domains described above, with equal weighting given to each. The second variable is a binary 
indicator derived from cluster analysis, a statistical methodology that creates a specified number 
of teacher groups based on the association of teacher responses to a series of items. We 
conducted separate cluster analyses for each of the above four domains. In each case, the derived 
clusters separated teachers into two groups: a high group that was high in reported practice in 
each domain and a low group, which represented those who reported a smaller sum of practice. 
We then conducted a second cluster analysis using the derived cluster variables. The final cluster 
variable is a binary variable (i.e., coded 1/0) that distinguishes two groups based on the 
individual cluster scores. The first group reflected teachers who were high on all the individual 
clusters and the second, teachers who were low on all the individual clusters. The two clusters 
thus represent teachers who more extensively implemented targeted practices in each domain 
(coded 1) versus those whose implementation was relatively less extensive. 

The teacher survey variables reflected four factors derived from an exploratory factor 
analysis. 1 This factor analysis included all survey items designed to parallel the log reports in 
four key domains: reading skills, writing skills, assessing student learning, and feedback. 
Exploratory factor analysis clustered items in four theoretically distinct factors which we 
characterized as attention to close reading of text, paragraph writing/structure, teacher-led 
formative assessment practice, and peer-oriented formative assessment practice. We then tested 
the reliability of the identified factors. As can be seen in Table 20, reliability was high for each 
of the factors, including Factor 4, which had relatively few items. Other survey variables include 
the total number of modules taught in 2011-2012 and 2012-2013 (a measure of teacher LDC 
experience), a measure of teachers’ commitment to teaching literacy in content area classes, a 


'Note that the exploratory factor analysis was performed on a larger sample of teachers that included eighth-grade 
history/social studies in Kentucky and Pennsylvania from our companion study (Herman et ah, 2015). Reliability 
analyses focused just on Florida teachers, and suggested that the constructs held for the smaller sample of teachers. 
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measure designed to capture teachers’ perceived capacity to teach LDC, perceived district and 
school support for LDC, the perceived utility of teacher collaboration around LDC, and a 
variable measuring the amount of professional development received. 

Table 20 


Reliability’ of Teacher Survey’ Factors 


Factor 

Description 

Number of items 

Cronbach’s alpha 

1 

Close reading 

10 

.87 

2 

Paragraph writing/structure 

4 

.84 

3 

Teacher-led formative assessment strategies 

11 

.82 

4 

Peer-oriented formative assessment strategies 

3 

.70 


Methodology 

The analysis started by classifying teachers as low, medium, or high value added, based on 
their students’ performance on FCAT 2.0. An HLM analysis encompassing the total population 
of Advanced Reading teachers and students and controlling for students’ prior performance and 
teachers’ prior effectiveness was used to establish cut-points for classification. The algorithm 
yielded essentially equal numbers of teachers in each of the three categories, and the cut scores 
were then used to classify the 52 teachers who completed logs and the 54 teachers who 
completed surveys as high, medium, or low value added. Mean scores on each implementation 
variable were then created for each group and differences tested using an ANOVA difference in 
means test. 

As the data in Table 21 and Table 22 show, for log variables, the high, medium, and low 
value-added groups were composed of 18, 15, and 19 teachers respectively, while the high, 
medium, and low value-added groups for survey variables consisted of 17, 20, and 17 teachers 
respectively. As with the implementation findings reported earlier, results show wide variation 
within each of the three groups. Essentially no differences were found to be statistically 
significant, which is not surprising given the relatively small sample sizes and the substantial 
within-group variation." 


‘Although one variable did show statistical significance, given the number of variables tested, we view this as a 
chance result. 
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Table 21 


Mean Scores on Teacher Log Implementation Variables for Teachers With Low, Medium, and High Value Added and ANOVA Test of Difference in Means 




Low value added 
(n = 1 9) 

Medium value added 
(n = 15) 

High value added 
(n = 1 8) 



Instmments 

Variable 

n 

Mean 

SD 

n 

Mean 

SD 

n 

Mean 

SD 

Test 

statistic 

p value 

Teacher log 

Range and intensity of reading 
instruction 

19 

0.60 

0.18 

15 

0.63 

0.26 

18 

0.55 

0.23 

0.60 

.55 

Teacher log 

Attention to close reading of 
text 

19 

0.27 

0.14 

15 

0.26 

0.23 

18 

0.23 

0.25 

0.17 

.85 

Teacher log 

Attention to basic reading skills 

19 

0.72 

0.23 

15 

0.76 

0.29 

18 

0.66 

0.26 

0.70 

.50 

Teacher log 

Attention to writing skills 

18 

0.87 

0.32 

15 

0.71 

0.28 

18 

0.82 

0.47 

0.78 

.46 

Teacher log 

Range and intensity of 
formative assessment of student 
learning 

19 

0.59 

0.30 

15 

0.53 

0.21 

18 

0.46 

0.29 

0.95 

.39 

Teacher log 

Range and intensity of feedback 
to students 

19 

0.52 

0.28 

15 

0.33 

0.16 

18 

0.35 

0.25 

3.49 

.04 

Teacher log 

Range and intensity of teacher 
literacy practices (reading skills, 
writing skills, formative 
assessment) 

19 

0.64 

0.22 

15 

0.55 

0.16 

18 

0.54 

0.27 

1.11 

.34 

Teacher log 

Teacher log cluster variable: 
high-quantity literacy practice 

18 

0.67 

0.49 

15 

0.40 

0.51 

18 

0.50 

0.51 

1.20 

.31 
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Table 22 


Mean Scores on Teacher Survey Implementation Variables for Teachers With Low, Medium, and High Value-Added and ANOVA Test of Difference in Means 




Low value-added 
(«=17) 

Medium value-added 
(n = 20) 

High value-added 
(»= 17) 



Instalments 

Variable 

n 

Mean 

SD 

n 

Mean 

SD 

n 

Mean 

SD 

Test 

statistic 

p value 

Teacher survey 

Factor 1: Attention to close 
reading of text 

16 

0.17 

0.84 

20 

0.45 

0.87 

17 

0.63 

0.91 

1.15 

.32 

Teacher survey 

Factor 2: Attention to 
paragraph writing/structure 

16 

0.24 

0.84 

20 

0.07 

1.04 

17 

-0.27 

1.00 

1.19 

.31 

Teacher survey 

Factor 3: Teacher-led 
formative assessment 
practice 

16 

-0.11 

0.93 

20 

0.18 

0.95 

17 

-0.02 

0.91 

0.45 

.64 

Teacher survey 

Factor 4: Peer-oriented 
formative assessment 
practice 

16 

0.07 

0.80 

20 

-0.09 

1.14 

17 

-0.04 

1.17 

0.10 

.91 

Teacher survey 

Total modules taught in 
2011-2012 and 2012-2013 
school years 

17 

7.00 

2.09 

20 

6.80 

2.26 

17 

8.35 

1.41 

3.23 

.05 

Teacher survey 

Support for teaching literacy 
in content area classrooms 

17 

2.02 

0.28 

20 

2.07 

0.34 

17 

2.14 

0.31 

0.62 

.54 

Teacher survey 

Teachers’ perceived capacity 
to teach LDC 

17 

1.96 

0.31 

20 

1.96 

0.32 

17 

1.88 

0.29 

0.38 

.69 

Teacher survey 

District and school support 
for LDC 

16 

1.34 

0.93 

18 

1.58 

0.75 

17 

1.59 

0.72 

0.50 

.61 

Teacher survey 

Utility of teacher 
collaboration 

16 

2.05 

1.02 

20 

2.10 

0.91 

17 

1.93 

0.65 

0.19 

.83 

Teacher survey 

Professional development 
dosage 

12 

2.00 

0.85 

16 

1.50 

0.73 

13 

1.69 

0.48 

1.74 

.19 
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Chapter 6: Summary and Conclusions 


This report has summarized CRESST’s study of the implementation and effects of LDC in 
sixth-grade Advanced Reading classes in a large, urban school district in Florida. The study is 
one of two conducted by CRESST, with funding from the Bill and Melinda Gates Foundation, to 
examine how LDC supports secondary teachers’ and students’ transition to the Common Core 
State Standards in English language arts. The Florida study features the use of LDC modules as 
core curriculum. Four centrally developed LDC modules, plus an orientation unit, provided the 
bulk of the district’s curriculum for sixth-grade Advanced Reading classes. 

In the sections below, we consider contextual factors that are important in interpreting 
study results before summarizing our findings with regard to the study’s primary evaluation 
questions: 

1 . How do teachers implement LDC? 

2. What is the impact of LDC on student learning? 

3. What conditions and contexts, including quality of implementation, influence LDC 
effectiveness? 

We conclude with implications and next steps for research and practice. 

Contextual Considerations 

The nature and generalizability of the study sample are a first, important contextual 
consideration. The study addresses only a subsample of those schools, teachers, and students 
across the country who are currently implementing LDC: the teachers and students in one large 
urban school district that piloted the intervention, and within this district, only sixth-grade 
teachers and students in Advanced Reading classes. The representativeness of the sample thus is 
an important constraint that limits the generalizability of any study findings. Demographically 
and in terms of prior achievement, the study sample looks similar to students in the state as a 
whole. However, the study cannot control for unobserved variables that may influence student 
success, and indeed, by virtue of their willingness to participate in early LDC trials, the district 
may well be unique. 

Further, to ensure that the study would not be attempting to judge intervention effects as 
teachers were initially learning how to implement it, the research focuses on teachers who had at 
least one year of prior experience with LDC. In fact, the majority of the teachers in the study had 
only one year of experience implementing the intervention prior to the study year, a prior year 


3 A companion study examines the implementation and effects of LDC in eighth- grade history/social studies 
and science in two states. See Herman et al. (2015). 
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that included both initial learning and initial implementation. Because the LDC implementation 
essentially involved implementing a new, yearlong curriculum composed primarily of four LDC 
modules, it is likely that teachers needed additional experience to be fully comfortable and expert 
in implementing it. The LDC modules, and the Common Core more generally, make new 
demands on teachers’ pedagogical and assessment practice, and therefore it is ambitious to 
expect LDC to have measurable impact on student learning at this early point in implementation. 

The sensitivity of available outcome measures is another limitation of the study. LDC is an 
intervention that involves students in close reading of fiction and informational text completing 
culminating evidence-based, extended writing assignments, which draw on the reading. Yet the 
FCAT state assessment — the only measure available for both LDC and comparison students — is 
a measure of reading comprehension that focuses primarily on fiction. The district writing 
measure, used in the study’s regression discontinuity design, also falls short in alignment with 
LDC in that it is not a reading-based prompt. Further, a major change in FCAT during the course 
of the study compromised the comparability from baseline to subsequent year scores for Phase 1 
schools that initially implemented LDC and may have confounded subsequent trajectories for all 
schools. 

Finally, the district that is the site of the study is known as a refonn leader, and LDC was 
one of several major district initiatives being implemented at the time of the study. For example, 
just prior to embarking on LDC, the district initiated a seven-year, $ 100-million effort to develop 
a new teacher evaluation system. The presence of multiple initiatives makes it difficult to isolate 
the effects of any one. 

How Did Teachers Implement LDC? 

Teacher logs and end-of-year teacher survey results indicate that study teachers did 
implement the major components of the intervention and followed the district modules in 
introducing module content and goals, engaging students in reading module texts, transitioning 
to writing, and working with students on their end-of-module writing assignment. The bulk of 
module time, as would be expected, was spent in the reading and writing components. In 
implementing these components, teachers reported developing their students’ skills in a range of 
reading and writing strategies. Furthermore, teachers reported frequent use of formative 
assessment: They noted using a variety of strategies to monitor their students’ ongoing learning 
in both reading and writing, and generally reported taking action when misunderstandings and/or 
problems were observed, again with substantial variation in the strategies used. 

The study’s analysis of district-developed LDC modules provides another window into the 
quality of LDC implementation. A specially developed assignment measure was used to assess 
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the quality of the district-developed modules on nine dimensions reflecting alignment with 
standards, quality and rigor of reading and writing tasks, quality of instructional strategies, 
fidelity to the LDC framework, and coherence. Quality was found to be moderate to high on all 
dimensions. The relatively highest ratings occurred for fidelity, coherence, instructional 
strategies, text alignment, and quality of the writing tasks. Alignment with Common Core 
English language arts goals and text rigor, while still in the moderate range, were more variable, 
which may well reflect a purposeful strategy to gradually transition students and teachers to the 
demands of the new standards. It is noteworthy that the district’s LDC modules were rated 
significantly higher in quality than those in the companion study of LDC implementation in 
eighth-grade content classes (see Herman et ah, 2015), which may suggest the value of involving 
literacy experts in module development. 

The sophistication of the modules and the variety of pedagogical strategies they 
incorporated, however, likely represented an implementation challenge for some teachers. 
Variation in teacher efficacy, as captured by teacher surveys, and in the quality of teachers’ 
implementation may provide one reason why the study did not find strong relationships between 
any single LDC implementation variable and student learning outcomes. That is, the relationship 
between LDC implementation and student learning may depend on how well teachers were able 
to implement specific strategies and respond to student needs in classroom interaction, among 
other unobservables. In time, as teachers gain additional expertise in the use of these strategies, 
discernible patterns may emerge. 

How Did LDC Affect Student Learning? 

Teacher perspectives. The implementation data suggest that teachers overall were 
committed to the LDC intervention. Survey results indicate that study teachers found LDC to be 
a helpful and effective tool in meeting a wide variety of instructional goals, including 
implementing the CCSS, incorporating formative assessment and teaching literacy into content 
area classes, and increasing the rigor of writing assessments. 

The majority of LDC teachers also agreed that their students experienced at least some 
success in each of the LDC component tasks — the reading mini-tasks, writing mini-tasks, and 
final writing task — and it is noteworthy that study teachers were more positive about their 
students’ success than were content teachers in the companion study. Although there was 
variation across respondents, teachers agreed at least somewhat that LDC had resulted in higher 
quality student writing, and supported students’ college readiness. 

CRESST ILA results. Students’ performance on the CRESST ILAs underscore teachers’ 
concern about LDC’s success for at least some of their students. The CRESST ILA generally 
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parallels the sequence of reading and writing activities in LDC: Students are asked to read and 
respond to several related texts about a central subject matter concept or topic and then to 
synthesize what they read with their existing knowledge to write an extended argumentative 
essay, explanatory essay, or other extended writing product. The study ILA focused on child 
labor, the topic of one of the district’s LDC modules. Student essays were scored on five 
dimensions, using a four-point scale: content understanding, rhetorical structure and quality, 
organization, use of evidence/text support, and grammar and conventions. 

The mean score for students across all dimensions was 2.3, indicating a basic level of 
performance. Scores for the content dimension were relatively the highest. Admittedly student 
motivation may have affected student scores, as the assessment was administered at the end of 
the year, but the scores also highlight the challenge of helping students achieve Common Core 
English language arts expectations for reading and writing. 

LDC impact on student learning. The study used quasi-experimental matched control 
group and regression discontinuity designs to examine LDC effects on student learning. The 
matched control group methodology used Coarsened Exact Matching (CEM) to identify a group 
of comparison students across the state of Florida, which was demographically and academically 
similar to the study LDC student group. Control students were selected based on student 
characteristics and prior achievement as well as the prior effectiveness of the schools they 
attended. 

Study analyses used both three-level longitudinal models and two-level cross-sectional 
models to test LDC effects on students’ FCAT scores in reading. Each modeled students’ 
performance under treated and non-treated teachers in sixth-grade Advanced Reading courses 
and included student demographic and prior achievement variables as covariates. The estimates 
thus controlled for observables in two ways, at both the matching and modeling stages. Results 
were consistent across all models, indicating the robustness of study findings: The analysis found 
no evidence of an overall LDC effect on FCAT reading scores. Interestingly, exploratory 
analysis of interaction effects suggests that initially higher achieving students may have 
benefited from LDC more than initially lower achieving students. 

In addition, a regression discontinuity design was used to test the effect of LDC on 
students’ scores on both the FCAT reading test and the study district’s writing assessment. Here 
the scores of students who were just below the prior achievement threshold for Advanced 
Reading placement were compared to those just above it, who were placed in Advanced 
Reading. We tested the effects of crossing the threshold for placement into Advanced Reading 
before and after LDC implementation. The comparison found no evidence of effects of 
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Advanced Reading placement on perfonnance on reading or writing assessments, either before 
or after LDC implementation. However, it is important to note that the design estimates the 
effect of the Advanced Reading placement (and by extension exposure to LDC) on the relatively 
lowest perfonning students in Advanced Reading, for whom large effects would not be expected. 
Results from the study’s matched control group analysis and a similar analysis in our companion 
report of LDC in eighth-grade history/social studies and science (see Herman et ah, 2015) both 
suggest that LDC effects are likely to be largest for relatively higher performing students. 

What Conditions and Contexts Influence LDC Effectiveness? 

Study analyses examining interactions between LDC treatment and student characteristics, 
in fact, further document such differential effects. In particular, the interaction of student prior 
achievement and LDC treatment was positive. That is, students who were relatively higher 
achieving prior to their LDC experience showed relatively greater benefit than did those who 
started relatively lower achieving, although the observed effect is small. No other interactions 
with student characteristics were statistically significant. 

Conclusions 

In summary, the study found no evidence of LDC effects on student learning, but it does 
provide lessons learned for teachers’ and students’ transitions to Common Core expectations. It 
also highlights important challenges involved in achieving and evaluating LDC success. 

A new Common Core-oriented curriculum. The study context provides a unique 
approach to the use of LDC: The LDC module design was used to create the nucleus of a 
yearlong curriculum aligned with the Common Core. Although not a central focus of the current 
study, this curriculum design model is a noteworthy accomplishment and would seem to provide 
a viable model for incorporating both the spirit and the specifics of the Common Core State 
Standards into instruction. Within and across modules, the curriculum exemplifies major shifts 
advocated in the new Standards, as well as core claims new standards-aligned assessments 4 are 
designed to measure, such as students’ ability to 

• read closely and analyze complex text, 

• write for a variety of audiences and purposes, and 

• build knowledge and conduct research through content-rich nonfiction. 


4 See Achieve the Core, Common Core Shifts for ELA 

(http://achievethecore.org/content/upload/122113_Shifts.pdf); Smarter Balanced Content Specifications for ELA 
(http://www.smarterbalanced.org/wordpress/wp- content/uploads/20 1 1/1 2/EL A-Literacy-Content- 
Specifications.pdf); PARCC assessment claims 

(http://www.parcconline.org/sites/parcc/files/FormattedItemGuidelines07.01.2013.pdf). 
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In building toward end-of-module writing performances that reflect these major 
competencies and claims, the modules also can be systematically designed to incorporate 
specific opportunities to learn and assess individual standards. This module-oriented approach 
seems nicely balanced between attention to specific standards and attention to a progression of 
learning that supports major student capacity, in authentic contexts that may support student 
motivation. The general approach seems an important step forward from the discrete, standard- 
by-standard approach to instruction that in the past has produced limited return. 

Positive effects on teachers. Despite the frequent adage that change is difficult and 
teachers are resistant to it, study data indicate that teachers were very positive about LDC and the 
major curriculum change it wrought. They had positive attitudes about the professional 
development they received and reported that they found LDC helpful and effective in meeting a 
variety of goals, including implementing CCSS, using formative assessment, incorporating more 
complex thinking and problem solving into curriculum and instruction, and improving student 
learning. Teachers’ reports about their fidelity of module implementation provide additional 
evidence of their positive attitudes. 

Struggles in moving to higher standards. With positive effects on teachers, the study also 
demonstrated the challenge of moving to more rigorous Common Core State Standards. We see 
evidence of this challenge in students’ basic performance on measures specifically designed to 
reflect the deeper learning demands of new standards and in teachers’ reports that sizable 
proportions of their students are struggling relative to the goals of LDC. 

That some teachers and students struggled is not meant to imply that current standards are 
unattainable or that Common Core expectations for students should be reduced — after all, we 
know that returning to prior standards will not get our children to 2 1st century success. However, 
the evidence does suggest that change will not come overnight and that both teachers and 
students will need support to meet the challenge. The issue is twofold: (a) How to address the 
needs and better prepare students and teachers who may not yet be ready to be successful with 
the challenges of LDC; and (b) how to modify and/or adapt the tools to scaffold teacher and 
student learning more effectively. 

Achievement gap implications. Although we regard findings of the interaction between 
student characteristics and LDC as tentative and subject to further validation, these results 
indicate that initially higher achieving students benefited more from LDC than did initially lower 
achieving students. Such a finding makes intuitive sense in that lower achieving students are 
more likely to have engaged in the “drill and kill” test preparation curriculum of the past, are 
least likely to have acquired the prior grade knowledge and skills expected by new standards, and 
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are least likely to have been engaged in the deeper conceptual understanding and applications 
that mark the new standards. However understandable, study findings thus suggest that, in the 
absence of additional scaffolding and supports for low-achieving students, LDC may be gap 
enhancing. 

Strengthening implementation. Although teachers reported implementing all module 
components, the findings suggest substantial variation in how they implemented the tool, and the 
sophistication of the strategies called upon in the modules suggest that at least some teachers 
may need more help. While district and school leadership supported implementation with 
common planning time and mentoring, and teachers found their colleagues very helpful, still 
teachers felt that they did not have adequate time to prepare nor to review and provide feedback 
on student writing. The former issue should dissipate with more experience with the curriculum, 
but the latter — time to analyze and provide feedback on student work — is more difficult to solve. 

The study did not achieve strong findings with regard to what aspects of implementation 
mattered most or what specific strategies were most effective. The findings are suggestive, 
however, of some factors that might be important in success: District support for LDC was clear 
across the sample, yet principal or local school support was more variable, suggesting a potential 
problem point. Teachers found their peers highly collaborative and helpful in implementing the 
two tools, but time for collaboration and more formal professional development was somewhat 
limited; investing more heavily in these supports may strengthen implementation. 

Challenges to discerning LDC effects. Some may consider study findings disappointing 
because they fail to reveal positive effects for LDC on student learning. It is well to remember 
the number of factors — previously noted — that clouded the study’s ability to detect such effects. 
The early stage of implementation is a first issue: study analyses were based on teachers who had 
only one year of prior experience in implementing the LDC-based curriculum. Given the extent 
of change this curriculum represented, it seems likely that teachers were not yet fully secure nor 
fully accomplished in implementing well the LDC-oriented approach. 

The sensitivity of study outcome measures is a second issue. The primary study measure 
was Florida’s state assessment, a selected response measure focusing on reading comprehension 
and one that gives relatively little attention to informational text. LDC, in contrast, focuses on 
reading and writing and particularly on writing from close reading of infonnational text. The 
district writing measure, a secondary measure used in study within-district analyses, also failed 
to align strongly with LDC goals. Further, by its nature, the latter analysis focused on the lowest 
ability students within LDC, a subpopulation on which LDC is least likely to have strong effects, 
based on other studies (see Herman et ah, 2015). 
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Other methodological challenges further confounded the study. The change in Florida’s 
state assessment starting in 2009-2010, the same year 10 Phase 1 pilot schools initiated LDC, 
produced a conundrum. Our analyses revealed that district Advanced Reading scores for both 
Phase 1 students and those not yet in LDC dropped precipitously relative to comparable students 
across the state, raising serious questions about the comparability of FCAT scores prior to and 
subsequent to the change. As a result, Phase 1 teachers and students had to be excluded from our 
matched control group quasi-experimental analyses. The pressure of responding to FCAT results 
also likely influenced Advanced Reading teachers’ subsequent teaching during the study year in 
unknown ways, particularly given that FCAT results were an element in school accountability 
ratings and teachers’ evaluation. A major effort to develop a new teacher evaluation system, in 
fact, was just one of many district reforms that occurred simultaneously with LDC 
implementation and that confound our ability to isolate LDC effects from the effects of other 
ongoing district and school initiatives. That is, these other initiatives may have supported or 
diluted LDC implementation and also contributed in unknown positive and/or negative ways to 
study learning outcomes. 

Concluding thoughts. In summary, our study findings of no evidence of LDC effect on 
student learning are simply that, and not evidence of no impact. The strength of study 
implementation findings — that teachers appeared to implement LDC as planned, generally felt 
positive and confident in its use, and believed that LDC is benefiting their students’ learning — 
would seem to indicate that a strong foundation has been established for strengthening and 
sustaining future LDC implementation. 

We leave it to future research to further examine LDC effects on learning under more 
sustained and longer tenn implementation as well as to examine issues of cost effectiveness. 
Future research and development also should continue the quest to identify both the most critical 
aspects of implementation in improving student learning and key infrastructure and supports that 
students who currently are struggling need to propel their success. 
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Appendix A: 

LDC Instruments and Rubrics 



Exhibit Al: 


LDC Teacher Log 

Thank you for taking the time to complete your teacher log for LDC instruction. This log 
should take no longer than five (5) minutes to complete. 

As a reminder, your logs will track instruction in only one of your LDC classes. Please 
remember to focus on the class period that was assigned to you when completing this log. 
For each log you complete, describe what happened in that particular class on that day . 


Is this your first log for this module? 

O Yes 
O No 

Is this your FINAL LOG for the current LDC module? 

O Yes 
O No 
O Not sure 


What are your primary goals for this module? 
Content goals 


Reading goals: 
Writing goals: 


How much total class time did students spend on any part of an LDC module today? 

0 minutes <10 minutes 10-30 minutes 30-50 minutes >50 minutes 


o 


o 


o 


o 


o 






Why was there no class time devoted to LDC today? 


□ I was absent 

□ School not in session 

□ Need to cover other content 

□ Field trip 

□ Other 


Check which aspects of the module students focused on today. (Check all that apply). If you 
are not sure how to characterize today’s lesson, check the aspect that best describes the focus 
of instruction. 

□ Preparing for the Task/Introducing Module 

□ Reading Process 

□ Transition to Writing 

□ Writing Process 


Consider the total amount of class time spent on module instruction today. What proportion 
of time was spent on the following classroom activities? (Total should add up to 100%) 


Lecture on subject matter content 

Mini-lesson on 

Explicit strategy instruction (e.g., 
teacher think-aloud, modeling, 
guided practice about a specific 
strategy) 

Whole-class discussion 
Small group work 
Pair/share 

Independent reading/writing 
Student presentations 
Other 





Preparing for the Task/Introducing Module 


How did you introduce the module? (check all that apply) 



Focus of 

Touched on 

Not today 


instruction 

briefly 


Overview and/or review of topic 

o 

o 

o 

Connect topic to students’ existing knowledge 

o 

o 

o 

Overview of readings 

o 

o 

o 

Review writing prompt 

o 

o 

o 

Review success criteria and/or rubric 

o 

o 

o 

Other 

o 

o 

o 


What did students do to prepare for module instruction? (check all that apply) 


Focus of 
student work 

Touched on 
briefly 

Not today 

Listened as I explained task 

o 

o 

o 

Came up with questions about topic 

o 

o 

o 

Made predictions about topic 

o 

o 

o 

Came up with questions about writing task 

o 

o 

o 

Re-wrote task in their own words 

o 

o 

o 

Brainstormed possible answers to prompt 

o 

o 

o 

Made predictions about reading 

o 

o 

o 

Reviewed exemplars of student work 

o 

o 

o 

Completed planning sheet (e.g., graphic organizer) 

o 

o 

o 

Discussed important strategies needed to complete task 

o 

o 

o 



Reading Process 


Which reading skills did students work on today? (check all that apply) 



Focus of 

Touched on 

Not today 


student work 

briefly 


Independent reading research 

o 

o 

o 

Making predictions/previewing 

o 

o 

o 

Summarizing important points 

o 

o 

o 

Note-taking/annotation 

o 

o 

o 

Identifying/ defining vocabulary 

o 

o 

o 

Analyzing text structure (e.g., how part relates to whole) 

o 

o 

o 

Interpreting information from graphical text 

o 

o 

o 

Distinguishing fact from opinion 

o 

o 

o 

Drawing conclusions from textual evidence 

o 

o 

o 

Citing textual evidence to support claims 

o 

o 

o 

Evaluating strength/weakness of evidence 

o 

o 

o 

Comparing arguments in two or more texts 

o 

o 

o 

Examining author's perspective/bias 

o 

o 

o 

Examining rhetorical devices 

o 

o 

o 

Other 

o 

o 

o 


Did you assess student learning of the skills listed above? 
O Yes 
O No 



If YES, to what extent did you rely on the following strategies to assess student 
understanding? (check all that apply) 



To a great 
extent 

To some 

extent 

Not at all 

Listened as students discussed text with peers 

o 

o 

o 

Circulated and reviewed student notes 

o 

o 

o 

Reviewed peers' feedback 

o 

o 

o 

Collected and reviewed student written responses and/or 
graphic organizers 

o 

o 

o 

Asked students to answer oral questions 

o 

o 

o 

Listened to students thinking aloud while reading 

o 

o 

o 

Led whole-class discussion 

o 

o 

o 

Listened to student questions 

o 

o 

o 

Assigned a quiz 

o 

o 

o 

Graded student work 

o 

o 

o 

Exit slips 

o 

o 

o 



If you did discover student misunderstanding about READING, to what extent did you rely 
on the following strategies to respond? (check all that apply) 



To a great 
extent 

To some 

extent 

Not at all 

One-on-one conference to provide feedback 

o 

o 

o 

Asked peer to provide feedback 

o 

o 

o 

Stopped class and modeled strategy 

o 

o 

o 

Wrote specific comments on student work 

o 

o 

o 

Scheduled in-class workshop time 

o 

o 

o 

Devoted time in lesson for students to use feedback 

o 

o 

o 

Grouped students together on a "need" basis for targeted 
instruction 

o 

o 

o 

Offered student a hint or suggestion 

o 

o 

o 

Gave student the answer 

o 

o 

o 

Gave student more time to try again and self-correct 

o 

o 

o 

Graded student work 

o 

o 

o 

Re-taught lesson segment 

o 

o 

o 

Planned to review skill in future lessons 

o 

o 

o 

Other 

o 

o 

o 


Transition to Writing 

How did you help students prepare for the writing task? (check all that apply) 



Focus of 
instruction 

Touched on 
briefly 

Not today 

Overview and/or review of topic 

o 

o 

o 

Review of readings 

o 

o 

o 

Review writing prompt 

o 

o 

o 

Review success criteria and/or rubric 

o 

o 

o 

Other 

o 

o 

o 



What did students do to prepare for the writing task? 



Focus of 

Touched on 

Not today 


instruction 

briefly 


Listened as I explained task 

o 

o 

o 

Came up with questions about topic 

o 

o 

o 

Made predictions about topic 

o 

o 

o 

Came up with questions about writing task 

o 

o 

o 

Re-wrote task in their own words 

o 

o 

o 

Brainstormed possible answers to prompt 

o 

o 

o 

Generated thesis statements 

o 

o 

o 

Reviewed exemplars of student work 

o 

o 

o 

Completed planning sheet (e.g., graphic organizer) 

o 

o 

o 

Generated essay outline 

o 

o 

o 

Selected relevant quotes from documents 

o 

o 

o 

Discussed important strategies needed to complete task 

o 

o 

o 



Writing Process 


What areas of writing did students work on today? (check all that apply) 



Focus of 

Touched on 

Not today 


student work 

briefly 


Generating ideas for writing 

o 

o 

o 

Outlining 

o 

o 

o 

Writing/text structure 

o 

o 

o 

Formulating a thesis statement 

o 

o 

o 

Formulating a counter-argument 

o 

o 

o 

Writing an introduction 

o 

o 

o 

Writing a conclusion 

o 

o 

o 

Writing a body paragraph 

o 

o 

o 

Using transitional words or phrases 

o 

o 

o 

Incorporating quotes/evidence 

o 

o 

o 

Style/word choice/syntax 

o 

o 

o 

Grammar conventions 

o 

o 

o 


Did you assess student learning of the skills listed above? 
O Yes 
O No 



If YES, to what extent did you rely on the following strategies to assess student 
understanding? (check all that apply) 



To a great 
extent 

To some 

extent 

Not at all 

Listened as students discussed draft with peers 

o 

o 

o 

Asked students to provide feedback to each other 

o 

o 

o 

Observed and reviewed student work 

o 

o 

o 

Collected and reviewed student writing exercises 

o 

o 

o 

Asked students to answer oral questions 

o 

o 

o 

Reviewed student rough drafts 

o 

o 

o 

Asked certain students to present writing to class 

o 

o 

o 

Assigned a quiz 

o 

o 

o 

Graded student work 

o 

o 

o 

Exit slips 

o 

o 

o 



If you did discover student misunderstanding about WRITING, to what extent did you rely 
on the following strategies to respond? (check all that apply) 



To a great 
extent 

To some 

extent 

Not at all 

Organized peer-editing session 

o 

o 

o 

Scheduled in-class workshop time 

o 

o 

o 

Held one-on-one conference with student 

o 

o 

o 

Devoted time in lesson for students to use feedback 

o 

o 

o 

Grouped students together on "need" basis for targeted 
instruction 

o 

o 

o 

Modeled skill using my own writing 

o 

o 

o 

Demonstrated skill using student's writing 

o 

o 

o 

Provided grammar mini-lessons 

o 

o 

o 

Wrote specific comments on student work 

o 

o 

o 

Had student revisit readings 

o 

o 

o 

Offered student a hint or suggestion 

o 

o 

o 

Gave student time to try again and self-correct 

o 

o 

o 

Corrected student writing 

o 

o 

o 

Graded student work 

o 

o 

o 

Re-taught lesson segment 

o 

o 

o 

Planned to review skill in future lessons 

o 

o 

o 

Other 

o 

o 

o 


How many weeks did you spend teaching this LDC module? 




How many days of instruction for each of the following? 
Preparation for the task 


Reading process 


Transition to writing 


Writing process 


Please list the approximate start date of Module 3: Communication in Cybernation, 
(mm/dd/yyyy) 


ADDITIONAL COMMENTS: 

We are very interested in your feedback. Please let us know if you have any questions or 
concerns about this log. Thank you! 









Exhibit A2: 

LDC Teacher Survey 2013 


[ log in from previous ] 


Before you begin, note that the Literacy Design Collaborative (LDC) Initiative goes by a number of different names. 

We use the phrases "LDC framework" or modules to refer to the tools that are part of this initiative. 

Some of the questions in the survey make reference to the Common Core State Standards (CCSS). In different states, this could be referred to 
differently, for example, in the state of Colorado, it is referred to as Colorado Academic Standards. 

The LDC Initiative is funded by the Gates Foundation. 

You are about to enter the survey. To go back a page, please use 
the survey's red "Back" button, not your browser's back button. 

Your answers will be saved each time you click "Next." 

The survey takes about 30 minutes to complete. You may leave and return multiple times. 

If you do return, after entering your login code, you will be placed in the screen you last visited. 

Please select the best answer for each question. Some instructions are in italics. 






TEACHER BACKGROUND INFORMATION 


1. What is / are your current position(s)? Please CHECK ALL that apply. 

□ a Classroom teacher 

□ b Reading specialist 

□ c Reading coach 

□ d Special education teacher 

□ e Librarian 

□ f Department head 

□ g Other (please specify) [ 100 characters ] 


2 . At which grade level(s) do you teach? Please CHECK ALL that apply. 

□ a Middle school (6 th - 8 th grade) 

□ b High school (9 th - 12 th grade) 


3. Which content areas do you teach? Please CHECK ALL that apply. 

□ a English/Language Arts 

□ b Science 

□ c Social Studies 

□ d Reading 

□ e Other (please specify) [ 100 characters ] 


4. To the nearest year, how long have you ... 

a) ... been a teacher? 

b) ... taught in your current school? 

c) ... taught in your current district? 


year(s) 

year(s) 

year(s) 


[ integer, 0-99 ] 
[integer, 0-99 ] 
[integer, 0-99 ] 





Yes 

No 

5. 

Do you teach ELL students? 

Ob 

O 0 

6. 

Do you teach special education students? 

O 

O 

7. 

Do you teach students who read or write below grade level? 

O 

O 

8 . 

Do you teach students with advanced literacy levels? 

O 

O 


PARTICIPATION IN LDC INITIATIVE 


9. 


How would you describe your participation in the LDC initiative? 

01 Required 

0 2 Voluntary 


0 3 I have not taught a module in 2012-2013. [ End survey; go to regular close ] 

0 4 I opted out of participating in 2012-2013 (please specify a reason for opting out) 


[ End survey; go to regular close ] 


[ 1000 characters ] 


10. Is this your first year in the LDC initiative? 

Oi Yes O 0 No 


11. My involvement with the LDC Initiative has included the following activities: (please CHECK ALL that apply) 

□ a Teaching a teaching task without a full module 

□ b Revising LDC modules that I did not develop myself 

□ c Developing LDC modules 

□ d Teaching LDC modules 

□ e Coaching others on how to use LDC modules 
□ f Presenting at an LDC professional development session 






MODULE DEVELOPMENT 


12. How many modules have you developed during the current school year (2012-13)? 


[ integer, 0-99 ] module(s) 


Please enter a 0 if you have not developed any modules during the current school year (2012-13). 


13. During the current school year (2012-13), I have ... 

Oi ... often 0 2 ... sometimes 0 3 ... never [skip if 12=0] 
...developed modules with the support of a colleague. 


14. How many modules have you revised during the current school year (2012-13)? 


[ integer, 0 - 99 ] module(s) 


Please include modules you developed in a previous year AND modules others developed. Enter a 0 if you have not revised any modules 
during the current school year (2012-13). 


15. During the current school year, I have ... 

Oi ... often 0 2 ... sometimes 0 3 ... never [skip if 14=0] 
... revised modules with the support of a colleague. 


[skip next if Q10 = yes ] 


16. H ow many modules did you teach last year (2011-12)? 


[integer, 0-99 ] 


module(s) 


Please enter a 0 if you did not teach any modules last year (2011-12). 


17. How many modules in total will you have taught during the current school year (2012-13)? 


[integer, 0-99 ] 


module(s) 




18. During the current school year (2012-13), I have ... 


Oi ... often 0 2 ... sometimes 0 3 ... never [skip if 17 = 0 ] 

...taught modules with the support of a colleague. 

19. Please indicate the frequency with which you are accessing existing modules on online? 
Oi ... often 0 2 ... sometimes 0 3 ... never 


SUPPORT FOR USING LDC MODULES 



20. Indicate whether the following people visited your classroom when you 

were teaching a 

module: 

[ note: randomize options, letters don't appear ] 

Visited 

Did not visit 

a. District or network LDC project lead 

o. 

O 0 

b. Principal 

o 

O 

c. Instructional coach/department head 

o 

o 

d. Teacher colleague 

o 

o 

BELIEFS ABOUT TEACHING LITERACY 




Q21 is about teaching literacy. 


21. Please indicate the degree to which you agree or disagree with the following statements: 

Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. 

Teachers from all content areas should help students improve their reading and writing skills. Oi 

o 2 

o 3 

o 4 

b. 

Science and social studies teachers do not have time to teach reading and writing. O 

o 

o 

o 

c. 

Writing assignments can help my students develop a deeper understanding of important 
concepts. O 

o 

o 

o 





PURPOSE OF INITIATIVE 


22. Please indicate the degree to which you agree or disagree with the statements below: 





Agree 

Disagree 




Agree 

Somewhat 

Somewhat 

Disagree 

The LDC framework is effective in ... 





a. 

... improving students' literacy skills. 

o. 

o 2 

0 3 

0 4 

b. 

... providing a curricular resource for teachers to address the Common Core State Standards. O 

o 

O 

O 

c. 

... encouraging science and social studies teachers to teach literacy skills. 

o 

o 

O 

O 

d. 

... encouraging secondary school teachers to teach literacy skills. 

o 

o 

O 

O 

e. 

... making instruction more engaging for the students. 

o 

o 

O 

O 

f. 

... using formative assessment to identify student strengths and weaknesses to inform 






instruction. 

o 

o 

O 

O 




TEACHER PERCEPTIONS OF TOOL UTILITY 


23. Please indicate whether using the modules has helped you in the following ways during module instruction: 


During LDC instruction, using the modules has helped me ... 

Yes 

No 

a. 

... find effective strategies for teaching my subject content. 

o. 

O 0 

b. 

... learn new ways to include formative assessment in my classes. 

o 

O 

c. 

... develop new ways to teach literacy skills in my content area. 

o 

O 

d. 

... learn detailed information about my students' literacy strengths and weaknesses. 

o 

o 

e. 

... provide students with more detailed feedback about their writing. 

o 

o 

f. 

... implement the Common Core State Standards. 

o 

o 

g- 

... increase the rigor of writing assignments. 

o 

o 

h. 

... better engage students. 

o 

o 




24. Please indicate the degree to which you agree or disagree with the statements below. 


LDC modules help me differentiate instruction ... 

f skip next if Q5 = no ] 

Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. ... for ELL students. 

[ skip next if Q6 = no ] 

o. 

o 2 

0 3 

o 4 

b. ... for special education students. 

[ skip next if Q7 = no ] 

o 

o 

O 

O 

c. ... for students who read or write below grade. 

f skip next if Q8 = no ] 

o 

o 

O 

O 

d. ... for students with advanced literacy levels. 

o 

o 

O 

O 






25. The modules are flexible enough to fit the needs of all my students. 

O 

o 

o 

o 

Questions 26a-d are about the most recent LDC module you tauaht. 

26. Please indicate the degree to which you agree or disagree with the statements below. 

Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. 1 knew what skills my students needed in order to complete the teaching task. 

o. 

o 2 

o 3 

o 4 

b. 1 knew the type of mini-tasks to give my students to prepare them to complete the 
template task. 

o 

o 

o 

o 

c. 1 understood how to use the LDC instructional ladder. 

o 

o 

o 

o 

d. Based on the information collected from using the LDC modules, 1 adjusted my 

instruction to meet the needs of individual students. 

o 

o 

o 

o 

27. Please indicate the degree to which you agree or disagree with the statements below. 

a. Using the modules raised my expectations for students' writing. 

o 

o 

o 

o 

b. The LDC framework has become an important part of my instructional practice. 

o 

o 

o 

o 


28. Select the phrase that best completes the following sentences: 
I use module instructional strategies ... 

Oi ... often 0 2 ... sometimes 0 3 ... rarely 


...during non-LDC instruction. 



STUDENT IMPACT 


29. Compared to my usual instruction, during the use of the modules, my students ... 

Oj ...are more engaged. 0 2 ...show the same level of engagement. 0 3 ...are less engaged. 


30. Please indicate the degree to which you agree or disagree with the statements below. 




Agree 

Disagree 



Agree 

Somewhat 

Somewhat 

Disagree 

a. The modules have resulted in higher quality student writing. 

o. 

o 2 

o 3 

o 4 

b. The LDC framework is supporting my students' college-readiness. 

o 

O 

O 

O 


Question 31a-b asks you to reflect back on your most recent experience implementing an LDC module during the current school year (2012-13). 

31a. When I taught the most recent LDC module, the majority of my students improved their understanding of content. 

Oi Yes O 0 No 

31b. When I taught the most recent LDC module, the majority of my students improved their literacy skills. 

Oi Yes O 0 No 



TOOL USE [FOI questions] 


Q32-37 are about your LDC instruction in the current school year (2012-13). 

32. What percent of class time did you spend on each of the following LDC components during your LDC instruction in the current school year 

(2012-13)? 


a. Preparation for Task/Introducing the Module 

b. Reading Process 

c. Transition to Writing 

d. Writing Process 

[Enter Number] 

[Enter Number] 

[Enter Number] 

[Enter Number] [limit the sum of the numbers in this ques to 100] 



33. Please indicate the degree of emphasis you placed on each of the following skills in your LDC reading instruction: 

[ note: randomize options, letters don't appear ] 

A great deal of emphasis Some emphasis Little Emphasis No Emphasis 


a. Independent reading/ research 

b. Making predictions/previewing 

c. Summarizing important points 

d. Note-taking/ annotation 

e. Identifying/ defining vocabulary 

f. Analyzing text structure (e.g. how part relates to whole) 

g. Interpreting information from graphical text 

h. Distinguishing fact from opinion 

i. Drawing conclusions from textual evidence 

j. Citing textual evidence to support claims 

k. Evaluating strength/ weakness of evidence 

l. Comparing arguments in two or more texts 

m. Examining author's perspective/bias 


Oi 

o 2 

o 3 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 

o 

o 

0 

0 


n. Examining rhetorical devices 



34. Please indicate the degree of emphasis you placed on each of the following skills in your LDC writing instruction: 
[ note: randomize options, letters don't appear ] 




A great deal of emphasis 

Some emphasis 

Little emphasis 

No emphasis 

a. 

Generating ideas for writing 

Oi 

o 2 

0 3 

o 4 

b. 

Outlining 

o 

o 

O 

O 

c. 

Writing/text structure 

o 

o 

O 

O 

d. 

Formulating a thesis statement 

o 

o 

O 

O 

e. 

Formulating a counter-argument 

o 

o 

O 

O 

f. 

Writing an introduction 

o 

o 

O 

O 

g- 

Writing a conclusion 

o 

o 

O 

O 

h. 

Writing a body paragraph 

o 

o 

O 

O 

i. 

Using transitional words or phrases 

o 

o 

O 

O 

j- 

Incorporating quotes/ evidence 

o 

o 

O 

O 



35. Please indicate how frequently you use the following strategies to assess student learning during your LDC instruction. [ note: randomize 


options, letters don't appear] 




Often 

Sometimes 

Rarely 

a. 

Listened as students discussed reading or writing with peers 

Oi 

o 2 

o 3 

b. 

Asked students to provide feedback to each other 

o 

o 

o 

c. 

Circulated and reviewed student notes and work 

o 

o 

o 

d. 

Collected and reviewed student writing exercises 

o 

o 

o 

e. 

Asked students to answer oral questions 

o 

o 

o 

f. 

Reviewed student rough drafts 

o 

o 

o 

g- 

Asked certain students to present writing to class 

o 

o 

o 

h. 

Assigned a quiz 

o 

o 

o 

i. 

Graded student work 

o 

o 

o 

j- 

Exit slips 

o 

o 

o 


Never 


Please indicate how frequently you use the following strategies to provide feedback to students during your LDC instruction 
[ note: randomize options, letters don't appear ] 

Often Sometimes Rarely 

a. 

Held one-on-one conference with student 

o. 

o 2 

o 3 

b. 

Asked peer to provide feedback or organized peer editing session 

o 

o 

o 

c. 

Stopped class and modeled strategy 

o 

o 

o 

d. 

Scheduled in-class workshop time 

o 

o 

o 

e. 

Wrote specific comments on student work 

o 

o 

o 

f. 

Offered student a hint or suggestion 

o 

o 

o 

g- 

Gave student more time to try again and self-correct 

o 

o 

o 

h. 

Gave student the answer 

o 

o 

o 

i. 

Graded student work 

o 

o 

o 

j- 

Re-taught lesson segment 

o 

o 

o 

k. 

Planned to review skill in later lessons 

o 

o 

o 

1 . 

Assigned grammar exercises 

o 

o 

o 


OOOOOOOOOO z 000000000000 





37. Please indicate the degree of success your students had in completing the following LDC activities? 


38. I 




A great deal of success 

Some success 

Little success 

No success 

a. 

The reading mini-tasks built into the instructional ladder 

o. 

o 2 

0 3 

0 4 

b. 

The writing mini-tasks built into the instructional ladder 

o 

o 

O 

O 

c. 

The final writing task 

o 

o 

O 

O 


i your most recent module, indicate yes or no for each of the following questions: [ note: randomize options, letters don't appear ] 


Yes 

No 

o. 

O 0 

O 

O 

O 

O 


a. I have used the LDC rubric to assess my students' final writing piece. 

b. I found the LDC rubric helpful in assessing my students' final writing piece. 

c. Using the rubric has helped my students understand the expectations for high quality writing. 




POTENTIAL BARRIERS TO TOOL USE 

39. Please indicate the degree to which you agree or disagree with the following statements. 







Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. 

1 had sufficient time to prepare to teach modules. 

o. 

o 2 

o 3 

o 4 

b. 

1 felt adequately prepared to effectively use modules. 

o 

o 

o 

o 

c. 

It is difficult to find the time to respond to student writing. 

o 

o 

o 

o 

d. 

1 am unsure about how best to give productive feedback to student writing. 

o 

o 

o 

o 

e. 

Using the LDC modules takes too much time away from covering required curriculum topics. 

o 

o 

o 

o 

f. 

It is challenging for me to find content-rich reading materials at my students' reading level. 

o 

o 

o 

o 

g- 

It is challenging for me to find the time to develop modules. 

o 

o 

o 

o 

SCALING of LDC INITIATIVE 


Q40 is about using the LDC modules next year ( 2013 - 14 ). 

40. Please indicate the degree to which you agree or disagree with the following statements 

a. I would like to develop modules next year. 

b. I look forward to teaching modules next year. 

c. I plan to improve how I teach modules next year. 


Agree Disagree 


Agree 

Somewhat 

Somewhat 

Disagree 

o. 

o 2 

o 3 

o 4 

o 

o 

o 

o 

o 

o 

o 

o 





41. Please indicate the degree to which you agree or disagree with the statements below: 


Agree Disagree 

Agree Somewhat Somewhat Disagree 


a. 

My participation in the LDC initiative is worth the time and effort involved. 

o. 

o 2 

o 3 

o 4 

b. 

1 see the ideas and practices of the LDC initiative gaining traction in my school. 

o 

o 

o 

o 

c. 

1 have noticed an increase in the number of teachers using the LDC modules/tasks in my 






school since last year. 

o 

o 

o 

o 

d. 

There are other curricular initiatives or programs in the district that address some of the 






same purposes as LDC. 

o 

o 

o 

o 

e. 

The other curricular initiatives or programs in the district create competing priorities 






with the LDC initiative. 

o 

o 

o 

o 

f. 

The district has the commitment to sustain the LDC initiative. 

o 

o 

o 

o 

g- 

The district has the funding to sustain the implementation of the LDC initiative. 

o 

o 

o 

o 


42. H ave you shared any of your LDC modules with a teacher who is not participating in the LDC initiative? Oi Yes O 0 No 

SCHOOL LEADERSHIP 

Q43 is about the administrators at your school. 




43. Please indicate the degree to which you agree or disagree with the following statements. 





Agree 

Disagree 


Don't 



Agree 

Somewhat 

Somewhat 

Disagree 

Know 

My school administrators ... 






a. 

... have a firm understanding of the LDC framework. 

Oi 

o 2 

0 3 

0 4 

O-99 

b. 

... have made formative assessment a priority at my school. 

o 

o 

O 

O 

O 

c. 

... encouraged me to participate in the LDC initiative. 

o 

o 

O 

O 

O 

d. 

... provided me with feedback about my instruction of the module(s). 

o 

o 

O 

O 

O 

e. 

... provided ongoing support for the implementation of the LDC tools. 

o 

o 

O 

O 

O 

f. 

... expressed concerns that teaching modules is taking time away 







from other instructional priorities. 

o 

o 

O 

O 

O 

g- 

... have attended professional development about the LDC framework. 

o 

o 

O 

O 

O 

h. 

... have communicated how the LDC framework is aligned with other 







school initiatives 

o 

o 

O 

O 

O 

District 

administrators ... 






i. 

... support the LDC framework. 

o 

o 

O 

O 

O 

j- 

... encourage my participation in the LDC initiative. 

o 

o 

O 

O 

O 

k. 

... provide ongoing support for implementation of the LDC framework. 

o 

o 

O 

O 

O 

1. 

... have a firm understanding of the LDC framework. 

o 

o 

O 

O 

O 

m. 

... have attended professional development about the LDC framework. 

o 

o 

O 

O 

O 



ALIGNMENT 

44. Please indicate the degree to which you agree or disagree with the following statements. 







Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. 

The LDC framework aligns well with my school's curriculum. 

Oi 

o 2 

0 3 

0 4 

b. 

The modules help prepare my students for current state assessment(s). 

O 

O 

O 

O 

c. 

The LDC framework aligns with the Common Core State Standards. 

O 

O 

O 

O 

d. 

1 see the unique value of the LDC framework to address the Common Core State Standards. 

O 

O 

O 

O 

e. 

The LDC rubric aligns well with my school's expectations for assessing student writing. 

O 

O 

O 

O 

COLLABORATION 


Q45-46 are about your interactions with your LDC colleagues. 


45. Do you and your LDC colleagues have regularly scheduled common planning time to discuss LDC? 
Oi Yes O 0 No 


Agree Disagree 

Agree Somewhat Somewhat Disagree 

Oi o 2 o 3 o 4 


46. I would describe my LDC colleagues as collaborative. 





47. About how often do you have scheduled meetings (as opposed to informal discussions) with your LDC initiative colleagues to discuss 
student work, instructional strategies, or teaching approaches? [skip if Q45=no] 

01 At least once a week 

0 2 Every other week 

0 3 Once a month 

0 4 Once per quarter/trimester/semester 

0 5 Never 


48. About how often do you have informal discussions (as opposed to scheduled meetings) with your LDC colleagues to discuss student work, 
instructional strategies or teaching approaches? 

01 At least once a week 

0 2 Every other week 

0 3 Once a month 

0 4 Once per quarter/trimester/semester 

0 5 Never 



Please indicate the degree to which you agree or disagree with the following statements. 
49. Collaboration with my LDC colleagues helps me ... 





Agree 

Disagree 




Agree 

Somewhat 

Somewhat 

Disagree 

a. 

... more effectively use the LDC framework. 

o. 

o 2 

o 3 

o 4 

b. 

... better support student learning. 

o 

o 

o 

o 

c. 

... develop LDC modules. 

o 

o 

o 

o 

d. 

... teach LDC modules. 

o 

o 

o 

o 

e. 

... revise LDC modules. 

o 

o 

o 

o 

f. 

... use the LDC framework rubric. 

o 

o 

o 

o 

g- 

... use students' products to inform my instruction. 

o 

o 

o 

o 

h. 

...provide helpful feedback to students about their writing. 

o 

o 

o 

o 


WORKING WITH EXPERIENCED LDC COLLEAGUES 


50. Are there teachers in your school or district who used LDC modules last year (2011-12)? 
Oi Yes 


O 0 No [skip to Q53] 


51. Did you work with a colleague more experienced with LDC this year (2012-2013)? 
Oi Yes 


O 0 No [ skip to Q53 ] 





52. H ow much did working with a colleague more experienced with LDC help you to develop and teach modules? 

01 a great deal 

0 2 a fair amount 

0 3 some 

0 4 not much 

0 5 not at all 


PROFESSIONAL DEVELOPMENT 


53. Have you participated in formal professional development sessions related to LDC during the current school year (2012-13)? 

Oi Yes O 0 No [ skip to 0.61 ] 


54. Which PD providers facilitated the LDC professional development you attended this year (2012-13)? Please CHECK ANY that applies 

□ a State or regional staff 

□ b External partner (e.g., Meta metrics, SREB, LDC) 

□ c District or network staff 

□ d School-based staff 

□ e I don't remember who facilitated the PD this year 

55. How many formal, scheduled LDC professional development sessions have you attended this year (2012-13)? [integer, 1-99 ] 




56. Please indicate whether you participated in the following types of LDC professional development sessions. 




Participated 

Did not pari 

a. 

One-on-one classroom visits 

o. 

O 0 

b. 

Coaching 

o 

O 

c. 

Webinars 

o 

o 

d. 

Small group meetings 

o 

o 

e. 

School-wide meetings 

o 

o 

f. 

District-wide meetings 

o 

o 

g- 

Cross-district meetings 

o 

o 


57. Please indicate whether the LDC professional development sessions you participated in was effective or not effective. 




Effective 

Not 

Effective 

a. 

[ fill choice from previous item here ] 

o. 

O 0 

b. 

[ fill choice from previous item here ] 

o 

o 

c. 

[ fill choice from previous item here, etc., etc. ] 

o 

o 


58. Please indicate whether the LDC professional development sessions you have participated in contained the following types of content: 


PD contained PD did not 

this content contain this content 


a. Using LDC modules as a way to implement the Common Core State Standards Oi 

b. Building a teaching task O 

c. Finding appropriate content materials O 

d. Designing modules O 

e. Using the instructional ladder O 

f. Using mini-tasks to address reading and writing skills O 

g. Providing students with feedback on their writing O 

h. Scoring student work with LDC rubric O 

i. Building modules with Module Creator O 

j. Differentiating module instruction to meet student needs O 

k. Implementing modules with special education students O 

l. Implementing modules with ELL students O 

m. Implementing modules with students who read or write below grade level O 

n. Implementing modules with students with advanced literacy levels O 


00000000000000 



0.59 asks about LDC professional development that would support your implementation of the LDC initiative. 


Please indicate whether you would like more LDC professional development on ... 

a) ...using LDC modules as a way to implement the Common Core State Standards 

Yes 

No 

b) 

... building a teaching task. 

o. 

O 0 

c) 

... finding appropriate content materials. 

o 

o 

d) 

... designing modules. 

o 

o 

e) 

... using the instructional ladder. 

o 

o 

f) 

... using mini-tasks to address reading and writing skills. 

o 

o 

g) 

... providing students with feedback on their writing. 

o 

o 

h) 

... scoring student work with the LDC rubric. 

o 

o 

i) 

... building modules with Module Creator. 

o 

o 

j) 

... differentiating module instruction to meet student needs. 

o 

o 

k) 

... implementing modules with ELL students. 

o 

o 

1) 

... implementing modules with special education students. 

o 

o 

m) 

... implementing modules with students who read or write below grade level. 

o 

o 

n) 

... implementing modules with students with advanced literacy levels. 

o 

o 

60. 

Are you compensated for attending professional development sessions? 

Oi Yes 

o 


No 



[ limit 1000 characters ] 
[ limit 1000 characters ] 


61. What additional supports and training would help you use the LDC framework? Please use the field below to describe. 


62. Surveys are not perfect. Maybe we missed some things that you think are important about the LDC initiative. Below, we invite you to 
write your assessment and comments about the framework as you have experienced it. 


63. What is your race/ethnicity? Please CHECK ONE that apply. 

□ a Native American 

□ b Asian/Pacific Islander 

□ c Black or African American 
□d Hispanic or Latino 
□ e White or Caucasian 
□ f Multiracial 

□ g Other (please specify) 

[ Go to "Regular Close" ] 





Thank you very much for the time and thought you have put into completing this survey. 


REGULAR CLOSE 


To ensure anonymity, your responses will be combined with those from teachers of numerous schools. 

Your responses will help to inform implementation of the Literacy Design Collaborative. 

DON'T AGREE CLOSE 

We are sorry you have chosen not to participate in the survey. 

Thank you for visiting Research for Action's and the National Center for Research on Evaluation, Standards, and Student Testing's survey on the 

Literacy Design Collaborative. 

ERROR MESSAGE IF AN ANSWER IS LEFT BLANK: 

You have not given an answer for a question on this screen. 

Do you want to go back to give an answer or continue with the survey? 

O I want to go back to answer the question. 


O I want to continue without answering the question. 




Exhibit A3: 

Rubric for LDC Module Implementation Measure 


Each dimension is scored on a 5-point scale ranging from "Fully Present or Realized" to "Not Present 
or Realized." 


Fully 

Sufficiently 

Moderately Present 

Barely 

Not 

Present or 

Present or 

or 

Present or 

Present or 

Realized 

Realized 

Realized 

Realized 

Realized 

5 

4 

3 

2 

1 


IMPORTANT: Descriptions are provided for three anchor points in the scale: 5 (Fully Present or 
Realized), 3 (Moderately Present or Realized), and 1 (Not Present or Realized). Use the intermediate 
points in the scale (4 and 2) to rate assessment practice that lies between 5 and 3 and 3 and 1. 


Dimension 1: Effective Writing Task 

Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 
Dimension 3: Text Alignment 
Dimension 4: Text Appropriateness 
Dimension 5: Text Rigor 

Dimension 6: Fidelity to LDC Module Instruction 
Dimension 7: Quality Instructional Strategies 
Dimension 8: Coherence and Clarity of Module 
Dimension 9: Overall Impression 




Dimension 1: Effective Writing Task 

Definition: Degree to which teaching task makes effective use of the template task's writing mode 
(i.e., argumentation or explanation); requires sustained writing and effective use of ideas and 
evidence to substantiate claims; and is feasible for most students to complete (i.e., appropriate for 
the grade-level and subject matter). 

Main Sources of Information: 

Module Creator Handout (Task) 

- Read and evaluate the teaching task, student background/prior knowledge, and summary 
information. 

- Evaluate the difficulty or ease students may encounter trying to answer the question. 

- Compare module teaching task to teaching task template options. 

5. Fully Realized 

The teaching task and performance expectations for the 
module are explicit and clear, require students to engage in 
higher-order thinking and writing, and are appropriate for the 
grade-level and subject matter. 

4. 


3. Moderately Present or Realized 

Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher- 
order thinking and writing and/or are not appropriate for the 
grade-level and subject matter. 

2. 


1. Not Present or Realized 

Minimal evidence of an effort to identify explicit and clear 
teaching task and performance expectations that provide 
opportunities for critical thinking and are appropriate for the 
grade-level and subject matter. 




Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 

Definition: Extent to which module addresses content essential to the discipline, as well as reading 
comprehension and writing standards informed by local and state standards. 

Main Sources of Information: 

Module Creator Handout (Task) 

- Read and evaluate the standards included in the module. 

- Module should include ELA as well as subject matter CCSS/state standards. 

- Compare and contrast the standards the module includes with those that could have been 
included. 

- Particular attention to content standards (CCSS History/Social Studies, Science, and Technical 
Subjects); State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 

5. Fully Realized 

Module specifically addresses content essential to CCSS and 
local or state standards in science or social studies, as well as 
reading comprehension and writing. All standards are well 
aligned to the topic and teaching task. 

4. 


3. Moderately Present or Realized 

Module broadly addresses content essential to CCSS and local 
or state standards in science or social studies and reading 
comprehension and writing. Standards are sufficiently aligned 
to the topic and teaching task. 

2. 


1. Not Present or Realized 

Minimal evidence that module addresses content essential to 
the discipline and literacy standards. Standards are poorly 
aligned to the topic and teaching task. 




Dimension 3: Text Alignment 

Definition: Degree to which assigned texts address teaching task content. 

Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Read and evaluate texts (hard copies or online). 

Student Work 

- References in student work. 

5. Fully Realized 

Assigned readings address the disciplinary content in science 
or social studies and give students the opportunity to gather 
information needed to complete the task. Readings are well 
aligned to the topic and teaching task, and provide students 
with well-balanced perspectives. 

4. 


3. Moderately Present or Realized 

Assigned readings mostly address the disciplinary content in 
science or social studies and give students some opportunities 
to gather information needed to complete the task. Readings 
are sufficiently aligned to the topic and teaching task, and 
provide students with moderately balanced perspectives. 

2. 


1. Not Present or Realized 

Minimal evidence that assigned readings address the 
disciplinary content in science or social studies and give 
students the opportunity to gather information needed to 
complete the task. Readings are poorly aligned to the topic 
and teaching task, and do not provide students with well- 
balanced perspectives. 




Dimension 4: Text Appropriateness 

Definition: Degree to which teaching task includes reading texts that are accessible to most students 
(i.e., appropriate for the grade-level and subject matter). 

Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Read and evaluate texts (hard copies or online). 

Student Work 

- References in student work. 

Anchor Readings 

- Read for examples of appropriate reading levels for 8 th grade students. 

5. Fully Realized 

Assigned readings are highly accessible and appropriate for 
most students in 8 th grade social studies or science classrooms. 
Selection of readings addresses the needs of students with a 
range of literacy skills, including students who are above, at, 
or below grade level, and English Language Learners. 

4. 


3. Moderately Present or Realized 

Assigned readings are mostly accessible and appropriate for 
the majority of students in 8 th grade social studies or science 
classrooms. Selection of readings sufficiently addresses the 
needs of students with a range of literacy skills. 

2. 


1. Not Present or Realized 

Assigned readings are not accessible or appropriate for 
students in 8 th grade social studies or science classrooms. 
Selection of readings poorly addresses the needs of students 
with a range of literacy skills. 




Dimension 5: Text Rigor 

Definition: Degree to which teaching task includes reading texts that use and develop academic 
understanding and vocabulary, and offer opportunities for multiple interpretations and higher-order 
thinking. 

Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Identify list of selected articles/links. 

- Read and evaluate texts (hard copies or online). 

- Consider issues of source credibility. 

Student Work 

- References in student work. 

5. Fully Realized 

Assigned readings require students to engage in higher-order 
thinking, and develop a strong academic understanding and 
vocabulary in social studies or science. Readings afford a deep 
conceptual and contextual understanding of the teaching task 
and topic. Selection of readings includes a broad range of 
credible primary and secondary sources. 

4. 


3. Moderately Present or Realized 

Assigned readings require students to engage in some higher- 
order thinking, and develop an adequate academic 
understanding and vocabulary in social studies or science. 
Readings afford a sufficient conceptual and contextual 
understanding of the teaching task and topic. Selection of 
readings includes a moderate range of credible primary and 
secondary sources. 

2. 


1. Not Present or Realized 

Assigned readings require students to engage in little higher- 
order thinking, or develop an academic understanding and 
vocabulary in social studies or science. Readings afford a 
limited conceptual and contextual understanding of the 
teaching task and topic. Selection of readings includes few 
credible primary and secondary sources. 




Dimension 6: Fidelity to LDC Module Instruction 

Definition: Degree to which module instruction , activities, and teaching task address each of the four 
stages of instructional practice (preparation for the task, reading process, transition to writing, 
writing process). 

Main Sources of Information: 

Module Creator Handout (Instruction) 

Information Sheet 

- Evaluate for distribution of activities and time spent on each of the four stages of instructional 
practice. 

5. Fully Realized 

The module instruction, activities, and teaching task reflect 
deliberate attention and fidelity to the four discrete stages of 
LDC module instruction. Classroom materials reflect 
demonstrable effort to develop instructional scaffolding within 
and across each stage of instruction. 

4. 


3. Moderately Present or Realized 

The module instruction, activities, and teaching task reflect 
moderate attention and fidelity to the four discrete stages of 
LDC module instruction. Classroom materials reflect sufficient 
effort to develop instructional scaffolding within and across 
each stage of instruction. 

2. 


1. Not Present or Realized 

The module instruction, activities, and teaching task reflect 
poor attention and lack of fidelity to the four discrete stages of 
LDC module instruction. Classroom materials reflect 
inadequate effort to develop instructional scaffolding within 
and across each stage of instruction. 




Dimension 7: Quality Instructional Strategies 

Definition: Degree to which the module provides clear instructional strategies aimed at helping 
students develop literacy skills and successfully complete the teaching task. And the degree to which 
module instruction and activities scaffold critical thinking and performance in a way that is 
meaningful within the context of a given field or subject-matter. 

Main Sources of Information: 

Module Creator Handout (Instruction) 

Classroom Handouts 

Student Work 

- Evaluate extent to which instructional strategies guide student learning in literacy and ability to 
complete the teaching task. 

- Evaluate extent to which the module activities scaffold critical thinking and student performance 
within the context of the subject matter at the core of the teaching task. 

5. Fully Realized 

Module provides clear and targeted instructional strategies 
and activities that scaffold student learning and promote 
critical thinking in social studies or science. There is explicit 
attention to helping students develop an accurate 
understanding of the topic and teaching task, and literacy 
skills necessary to successfully complete the writing task. 

4. 


3. Moderately Present or Realized 

Instructional strategies and activities are available to support 
adequate student learning and critical thinking in social 
studies or science. There is moderate attention to helping 
students develop an understanding of the topic and teaching 
task, and literacy skills necessary to complete the writing task. 

2. 


1. Not Present or Realized 

Limited instructional strategies and activities available to 
support student learning and critical thinking in social studies 
or science. Insufficient attention to helping students develop 
an understanding of the topic and teaching task, or literacy 
skills necessary to complete the writing task. 




Dimension 8: Coherence and Clarity of Module 

Definition: Degree to which there is logical alignment between the teaching task and other module 
goals with readings, mini-tasks, and instructional strategies. 

Main Sources of Information: 

Module Creator Handout 

Classroom Handouts 

Student Work 

5. Fully Realized 

Strong alignment between the teaching task and goals of the 
module, including the CCSS and local and state literacy and 
content standards, with the readings, mini-tasks, student 
work, and instructional strategies. 

4. 


3. Moderately Present or Realized 

Moderate alignment between the teaching task and goals of 
the module, including the CCSS and local and state literacy and 
content standards, with the readings, mini-tasks, student 
work, and instructional strategies. 

2. 


1. Not Present or Realized 

Poor alignment between the teaching task and goals of the 
module, including the CCSS and local and state literacy and 
content standards, with the readings, mini-tasks, student 
work, and instructional strategies. 




Dimension 9: Overall Impression 
Definition: Holistic assessment of LDC Module. 

Main Sources of Information: 

Module Creator Handout 
Classroom Handouts 
Student Work 

Main question: To what extent does this module contribute to student college readiness and 
development of advanced literacy skills? 

5. Advanced LDC Module Implementation 

4. Proficient LDC Module Implementation 

3. Adequate LDC Module Implementation 

2. Marginal LDC Module Implementation 

1. Inadequate LDC Module Implementation 
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ENGLISH LANGUAGE ARTS 


Assessment Directions 

This is a two part assessment. In part 1 , you will go over the directions, read the materials, and 
answer questions. In part 2, you will write an essay on the topic provided. 

You will be asked about the rights of children and how they apply to child laborers. 

You may take notes directly on the assessment. 

The materials include: 

1. Reading Passage - Rizwan, Pakistan 

2. Reading Passage - Shiv, India 

3. Reading Passage - United Nations Convention on the Rights of the Child 

Underlined terms are defined in a glossary. 

Stop at the sign at the end of part 1. 
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1. Reading Passage - Rizwan, Pakistan 

The passage below includes excerpts from a story about a teenager in Pakistan. The story comes 
from a larger collection of interviews put together by The International Center on Child Labor and 
Education (ICCLE). 


Rizwan, 14, is from a village near Lahore in Pakistan called Chachey Wali. He is the third 
oldest child of eight children in his family. He has three brothers and four sisters. 

Rizwan’s mother has not been well since she received an electrical shock four years ago. His 
father is unable to find a job. Because of this, Rizwan and four of his siblings (ages 5-13 
5 years) are forced to work making bricks. Rizwan’s 3 year-old brother and two oldest sisters 
do not work. The sisters, who are 16 and 18 years old, are married. In Rizwan’s society, it is 
out of the question for them to go out to work. 

In Chachey Wali, there are twelve brick kilns . Rizwan started working in the kilns at the age 
of seven. Rizwan gets up at 4:00 a.m. and makes bricks for eleven hours a day, six days a 
10 week. He works all day long outside in the hot sun. Rizwan earns 500 Pakistan rupees 
($8.35) a week for making about 1,000 bricks. 

Brick-making is tough work. Bricks are made outdoors, meaning that the children who make 
them have to sit in the harsh sun all day. Most people who make bricks have to sit on their 
heels and bend forward, hurting their spine, neck, feet and knees, causing extreme 
15 discomfort. The most common side effects of this work are back pains, skin infections, and 
lung infections from inhaling a lot of dust. Because of the long hours and tough conditions, 
Rizwan does not think he and his siblings will be able to work much by the time they are 25 
years old. 

Rizwan does not like his bosses, who yell at him and embarrass him and his siblings when 
20 they make mistakes. Once, a supervisor hit his younger sister for laying her foot on a brick 
before it dried. Rizwan has also been injured at work. He has a scar on his forehead from a 
nail. When he is sick and cannot work, Rizwan does not get paid. 

Through a human rights organization called Godh Lahore, Rizwan receives informal 
education for two hours each night, six nights a week. So far, he has completed four years of 
25 school. 

Adapted from http://www.knowchildlabor.org/tme_stories/pdf/ICCLE_RIZWAN_Pakistan.pdf 


Glossary 


kiln - an oven for drying bricks 

siblings - brothers and sisters 

rupees - money in Pakistan 
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Reading Comprehension Questions 

In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 

Choose the best answer for each question. Fill in the bubble next to the answer you choose. 

1. What does the author mean by “Rizwan and four of his siblings (ages 5-13 years) 
are forced to work” (lines 4-5)? 

@ All children five years or older are required to work in Chachey Wali. 

® The bosses made Rizwan and his siblings work. 

© Rizwan’s father wanted them to work. 

© The children had no choice but to work to support their family. 

2. What educated guess can you make about Rizwan’s society from the following 
sentence: “In Rizwan’s society, it is out of the question for them [Rizwan’s two 
older sisters] to go out to work” (lines 6-7)? 

® Women are not considered strong enough to work in the kilns. 

© Teenagers who are 16 years or older are not allowed to work. 

© Women are not allowed to work once they are married. 

© Married women do not need to work. 

3. According to the article, which is true of brick-making? 

® It pays more money than farming. 

© It can lead to physical problems. 

© It requires being inside all day. 

© Only children are allowed to work in the kilns. 
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Short Answer 

4. The reading passage discusses the conditions of working in brick kilns (lines 12- 
18). Based on the information in the passage, list two reasons why employers want 
to hire children like Rizwan instead of adults to work in brick-making. 


a) 


b) 
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2. Reading Passage - Shiv, India 

The passage below includes excerpts from a story about a teenager in India. The story comes from 
a larger collection of interviews put together by The International Center on Child Labor and 
Education (ICCLE). 


Shiv, 16, is from Madhepura village in the state of Bihar in India. He has three brothers: 8, 
13, and 15 years old, and two sisters: 18 and 20 years old. Both Shiv’s mother and father 
work as agricultural laborers. All of his brothers and his sisters help their mother when she 
works on the farm. None of them goes to school, as there is no school in the village. 

When he was barely seven years old, Shiv was lured away from home by a man who 
promised him chocolate. The man took him 370 miles away from his home to the Varanasi 
district in India, where he worked in a carpet factory for five years around the clock for no 
pay. He was abused and beaten by his employer. "Have you ever realized that carpets are 
made by children like me?" Shiv asked. Since he worked for no pay and was not allowed to 
leave the factory, Shiv was considered to be a bonded laborer. 

"The work was hard," Shiv said. He worked 16 hours a day from 4:00 a.m. to 9:00 p.m., 
seven days a week with an hour for lunch. He used to get up at 4:00 a.m. and go to bed at 
10:00 p.m. He did not go to school. 

Shiv lived, slept, and ate in the same room where he worked. He lived and worked with 1 8 
other boys between the ages of 7 and 16. Shiv was not given proper food. The children did 
not get breakfast. For lunch and dinner, they received "very badly cooked" rice and lentils. 

There was no time to play or have fun. Shiv was not allowed to go outside, even though it 
was very hot and dirty inside and there was no fan. Shiv was often sick and tired but was 
never taken to the doctor. He was just given some pills and told to continue working. Often, 
he cut his fingers while weaving. When this happened, the employer used a chemical to bum 
shut the cuts on his fingers. He said it was very painful. If Shiv ever said "no" to work, he 
was subjected to verbal and physical abuse. 


r i 

continue reading 

L A 


Glossary 


lured - tempted 


whim - sudden desire or change of mind 
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Shiv did not like his boss, who only wanted the boys to work hard and produce more every 
day. "For this he used to beat us. His behavior was very bad with us. He used to ill-treat us. 
He always used bad language with us. If there was any small mistake, or when we did not 
meet the deadline for completing the carpets, we were beaten very badly," Shiv said. This 
happened quite often, at the employer’s whim . His boss never praised anyone. 

For five years, Shiv worked without any time off. He never went home or saw anyone in his 
family. He did not enjoy a single day of work. At that time he had no dreams. 

After five years in the factory, Shiv was rescued by Mr. Kailash Satyarthi and a team of human 
rights activists when local police raided the factory where Shiv was working. Now Shiv lives and 
attends school in New Delhi. "Now I am getting love and education," Shiv said. He goes home 
during school vacations. Shiv has completed the fifth grade and no longer works. 


Adapted from http://www.knowchildlabor.org/tme_stories/pdf/ICCLE_SHIV_India.pdf 
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Reading Comprehension Questions 

In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 

Choose the best answer for each question. Fill in the bubble next to the answer you choose. 

1. If Shiv had not been taken to the factory, how would he have probably have spent 
his childhood? 

© going to school 
® playing and having fun 
© making carpets 
© working on the farm 

2. Based on the passage, you can infer that a bonded laborer is defined as: 

© a child taken against their will. 

® someone who works for no pay and has no freedom. 

© a child who is beaten by their employer. 

© someone who works long hours. 

3. What was the main goal of the carpet factory boss? 

© to keep the boys healthy so they could work more 
© to abuse the boys physically 
© to produce many carpets quickly 
© to produce high-quality carpets 
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Short Answer 

4. In 1-2 sentences, explain why Shiv “had no dreams” while he was working in the 
factory. 
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3. Reading Passage - United Nations Convention on the Rights of the Child 

The articles below are taken from the United Nations Convention on the Rights of the Child , which 
is a human rights treaty that lists the civil, political, economic, social, health and cultural rights of 
children. 


Nations that sign this document are required by international law to follow it. The Convention has 
been signed by 193 countries, including India and Pakistan. There are a total of 54 articles in the 
original document. 


Article 1: Everyone under 18 has all the rights listed in this document. 

Article 3: All adults should always do what is best for you. 

Article 5: You have the right to be given guidance by your parents and family. 

Article 9: You have the right to live with your parents, unless it is bad for you. 

Article 15: You have the right to meet with friends and to join groups and organizations, as 
long as it does not stop other people from enjoying their rights. 

Article 19: You have the right to be protected from being hurt or badly treated. 

Article 24: You have the right to the best health possible and to medical care and to 
information that will help you stay well. 

Article 28: You have the right to education. 

Article 35: No one is allowed to kidnap you or sell you. 

Article 36: You have the right to protection from any other kind of exploitation . 

Adapted from "Convention on the Rights of the Child." U.N. Security Council, 44 th Year (A/RES/44/25), Official Record, November, 1989, 
pp. 166-173. 


Glossary 


Article - a section of a legal exploitation - taking unfair advantage of 

document someone 
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Reading Comprehension Questions 

In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 

Choose the best answer for each question. Fill in the bubble next to the answer you choose. 

1. Why are Rizwan and Shiv protected by the Convention on the Rights of the Child ? 

© They don’t live with their parents. 

© They are younger than 18. 

©) They both work in factories. 

© They were both kidnapped and taken from their families. 

2. Which of the following is a violation of Article 24? 

© Shiv’s boss used bad language and swore at the workers. 

® Rizwan’s boss hit his sister for making a mistake. 

© Shiv’s boss burned his cuts and did not take him to the doctor. 

© Rizwan’s boss embarrassed him in front of the other workers. 

3. Why might Shiv’s boss oppose Article 15? 

© He would not want Shiv to play sports and become strong. 

® He would not want Shiv to learn about his rights. 

© He would not want Shiv to have any friends. 

© He would not want Shiv to enjoy his free time. 
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ENGLISH LANGUAGE ARTS 


Short Answer 

4. a. List one Article violated by Rizwan’s boss. 


Explain your answer. 


b. List one Article violated by Shiv’s boss. 


Explain your answer. 
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ENGLISH LANGUAGE ARTS 



End of Part 1 
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ENGLISH LANGUAGE ARTS 



Writing Task 

Write a formal essay explaining three ways in which Shiv’s and Rizwan’s bosses violated the boys’ 
rights according to the United Nations Convention on the Rights of the Child. 


Be sure to support your explanation using examples from the readings and what you have learned 
about child labor. 


Your essay will be scored on how well you: 


• demonstrate an understanding of the issue of child labor and the United Nations Convention 
on the Rights of the Child. 

• state and support your thesis with information from the readings. 

• present your essay in a logical and well-organized manner. 

• use the materials to support your explanation. 

• use proper punctuation, spelling, and grammar. 
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Exhibit A5: ILA Writing and Short Answer Rubrics 


CRESST Literacy Assessment Scoring Rubric for Grade 6 
Overview 

The five scoring dimensions are: content understanding, rhetorical structure/quality, reference to text, and 
use of grammar/conventions. There may be some overlap between some of these dimensions, which is fine. 
When scoring, keep in mind we are looking at on overall indicator of proficiency on each of these 
dimensions. 


Dimension 

Name 

Description 

A 

Content understanding 

This is a measure of overall how well the student has 
demonstrated that they understand the materials and the 
topic in their essay. 

B 

Rhetorical structure/quality 

Explanation: Measures how well the elements of 
explanatory writing as described in the response. For 6 th 
grade writing, explanations should establish a thesis and 
develop the topic with information, examples, and analysis. 

C 

Organization 

This evaluates the focus, logical progression of ideas, and 
structure demonstrated by the student's writing. 

D 

Reference/support with text 

This is a measure of how well references to text details are 
used to support statements in the. A text detail is a 
quotation, paraphrase, or any other reference to information 
and ideas in the texts provided. 

E 

Grammar and Conventions 

Evaluates the command of standard English conventions 
demonstrated by the response: proper English usage and 
control of grammar, formal tone, correct paragraph and 
sentence structure. 
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Student ID: 


Score point 

Explanation Rubric 


Score 

Content 

understanding 

The response demonstrates well-developed and thorough understanding 
of the topic. 

4 

□ 

The response demonstrates solid understanding of the topic. 

3 

□ 

The response demonstrates some understanding of the topic. 

2 

□ 

The response demonstrates little or no understanding of the topic. 

1 

□ 

Rhetorical 

structure/quality 

Important elements of the explanation are clearly and thoroughly 
described and articulated and the response is aligned to the prompt. 

4 

□ 

Elements of the explanation are clearly described and the response is 
aligned to the writing prompt. 

3 

□ 

There is an attempt to describe some elements of the explanation. 

2 

□ 

Elements of the explanation are not described, or the descriptions are 
unclear. 

1 

□ 

Organization 

The essay is well organized. 

4 

□ 

The essay is appropriately organized. 

3 

□ 

The essay is somewhat organized. 

2 

□ 

The essay has little or no appropriate organization. 

1 

□ 

Reference 
support with 
text 

The response uses detailed and well chosen references to the text to 
thoroughly develop the explanation. 

4 

□ 

The response uses accurate and detailed references to the text to provide 
solid development of the explanation. 

3 

□ 

The response uses some accurate and detailed references to the text to 
develop the explanation. 

2 

□ 

The response uses little or no accurate and detailed references to the text 
to develop the explanation. 

1 

□ 

Grammar and 
Conventions 

The response demonstrates a well-developed command of standard 

English conventions. 

4 

□ 

The response demonstrates a solid command of standard English 
conventions. 

3 

□ 

The response demonstrates some command of standard English 
conventions. 

2 

□ 

The response demonstrates little or no command of standard English 
conventions. 

1 

□ 
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Child Labor Short Answer Rubric 


Child Labor 

Question ID 

Question 

Question Stem & Sample 

Answer 

Scoring Rubric 

MLA_RP_133 

4 

The readings passage discusses 
the conditions of working in 
brick kilns. Based on the 
information in the passage, list 
two reasons why employers 
want to hire children like 

Rizwan instead of adults? 

Answer: a) Brick makers have to 
sit low to the ground, and 
children are small and can do 
this better. 

b) the owners can intimidate and 
control the children easier than 
adults. 

BL = Blank 

0 = The student gives either an 
incomplete or incorrect response. If 
reasons are given, they are not based 
on the article. 

1 = The student provides one reason 
for why children are hired to work as 
brick makers. The reason given is 
both plausible and based on the text, 
e.g. “they have small hands and work 
faster” is not mentioned in the article. 

2 = The student provides two reasons 
that are based on the article. Ideas 
mentioned in the article that could be 
used in reasoning include but are not 
limited to: 

• low pay 

• abuse and intimidation 

• physical challenges of brick 
making 

• long term damage to 
health/ability to work 

MLA_SI_113 

4 

In 1-2 sentences, explain why 

Shiv “had no dreams” while he 
was working in the factory. 
Answer: Shiv was taken away 
when he was very young; he 
worked long hours, and was not 
allowed any free time. He had 
very little hope for the future. 

BL = Blank 

0 = The student gives either an 
incomplete or an incorrect response. 

If given, the explanation is inaccurate 
or literally addresses sleeplessness, 
not dreaming while sleeping, not day 
dreaming, etc. 

1 = The student provides at least one 
reason why Shiv had no hope for his 
future. E.g. 

• Describes his life/living 
conditions. 

• Describes his state of mind as a 
result of his living conditions. 
(This does not require details, as 
long as it is a reasonable 
conclusion.) 
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Child Labor 

Question ID 

Question 

Question Stem & Sample 

Answer 

Scoring Rubric 

MLA_UNC_1 

17 

4, part a 

List one Article violated by 
Rizwan’s boss: Explain your 
answer. 

Answer: Student answers will 
vary, but the article should relate 
to Rizwan’s situation and the 
explanation should support and 
explain the reasoning behind 
why they selected that particular 
article. 

BL = Blank 

0 = The student gives either an 
incomplete or an incorrect response. 

1 = The student lists a correct article 
but fails to explain their answer. 
Example: Article 19 

2 =The student gives both a correct 
article and explains why they selected 
that article. 

MLA_UNC_1 

19 

4, part b 

List one Article violated by 

Shiv’s boss: Explain your 
answer. 

Answer: Student answers will 
vary, but the article should relate 
to Shiv’s situation and the 
explanation should support and 
explain the reasoning behind 
why they selected that particular 
article. 

BL = Blank 

0 = The student gives either an 
incomplete or an incorrect response. 

1 = The student lists a correct article 
but fails to explain their answer. 
Example: Article 35 

2 =The student gives both a correct 
article and explains why they selected 
that article. 
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Developing an Assignment Measure of Common Core State Standards Literacy Practice 


The National Center for Research on Evaluation, Standards and Student Testing (CRESST) is 
developing evaluation tools to support the transition to the Common Core State Standards (CCSS). 
Generously supported by the Bill and Melinda Gates Foundation, the CRESST tools include measures of 
both teacher practice and student learning. The Assignment Measure tool presented here examines how 
well 8th grade science and social studies teachers were able to incorporate selected CCSS standards in 
English Language Arts into classroom curriculum and instruction. The tool specifically measures how 
teachers implemented the Literacy Design Collaborative (LDC) framework, an intervention designed to 
help teachers create integrated literacy and content instruction around core writing tasks. 

Assignment and artifact collection has been identified as an efficient and economical means of 
measuring the quality of classroom instruction (e.g., Clare & Aschbacher, 2001; Newmann, Bryk, & 
Nagaoka, 2002; Storms, Riazantseva, & Gentile, 2000; Matsumura, Slater, Wolf, Crosson, Levison, 
Peterson, Resnick, Junker, 2006; Martinez, Borko, Stetcher, Luskin, & Kloser, 2012). Classroom 
assignments shed light on such variables as teacher clarity, cognitive rigor of instruction, and in this case, 
degree of LDC fidelity of implementation. Careful assessment of assignments can potentially capture 
instructional quality with a degree of accuracy that approaches actual observation. In addition to shedding 
light on the nuances of instructional practice, artifact collection and analysis can potentially be leveraged 
for professional development purposes, to model and improve effective practices. The CRESST 
Assignment Measure was designed with both of these uses-instructional evaluation and professional 
development — in mind. Furthermore, the benefit of the Assignment Measure is its transferability: with 
only minor modifications, the measure can be used to assess to any instruction that involves reading texts 
and writing about them. 

In contrast to previous artifact studies that examine discrete assessment practices or assignments 
(cf. Martinez, Borko, Stetcher, Luskin, & Kloser, 2012), the CRESST Assignment Measure was designed 
to capture instruction that occurred within the framework of an LDC “Module,” a unit of instruction 
lasting 2-4 weeks that culminates in a summative writing task. In building an LDC Module, teachers 
begin with fill-in-the-blank template tasks — or extended writing prompts — that are designed to help them 
incorporate the literacy demands specified in the Common Core State Standards into units of instruction 
in literature, history/social studies, or science. Teachers design instructional activities using the LDC 
framework that is comprised of four “skill clusters:” (1) Preparing for the Task, (2) Reading Process, (3) 
Transition to Writing, and (4) Writing Process. Each LDC Module includes an instructional ladder of 
“mini-tasks” that build the requisite skills in both reading and writing to complete the final writing task. 
The final product — instructional ladder plus template task — is referred to as an LDC module. 
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LDC has also designed Module Creator, an online platform that walks teachers through the four 
steps of building a module: What task? What skills? What instruction? What results? This platform is 
replete with pull-down menus and default settings that teachers are invited to augment and/or revise, 
depending on their instructional needs. For example, in the Reading Process skills cluster, teachers are 
invited to add additional skills beyond the ones suggested by LDC: text selection, active reading, essential 
vocabulary, academic integrity, note-taking. Likewise, if they choose to teach the skill of “active 
reading,” they have the option of adding additional instruction or “mini-tasks” beyond the default option 
on Module Creator, which simply has students “brainstorm ways to figure out any author’s intent” and 
“share and discuss their answers for each text.” 

The CRESST Assignment Measure was designed to assess both the clarity and quality of the 
writing task and the appropriateness and relevance of the activities in the instructional ladder. Despite the 
fill-in-the-blank format of the final writing task, the process of building an LDC module leaves 
tremendous room for variation. Teachers are responsible for choosing the topic, selecting appropriate 
texts, and designing daily instruction around all elements of literacy, from reading comprehension, 
paragraphing, to all aspects of writing style and mechanics. This potentially wide range in teacher 
implementation has important implications for the design of fidelity of implementation measures. The 
breadth of instruction contained in a given LDC Module poses logistical challenges for artifact collection, 
which will be discussed below. 

In the report below, we describe the CRESST Assignment Measure and report findings from a 
reliability study. In particular, we examine variation and sources of error in ratings, attending to what 
these results might suggest about teacher implementation of LDC. Lastly, using both the quantitative 
results from the reliability study, as well as qualitative data from interviews and surveys, we make 
suggestions for how the Assignment Measure might be used in the future. 

CRESST Assignment Measure 

CRESST’s LDC Assignment Measure was designed to capture the specific demands of 
integrating content and literacy, as is required in subject area classrooms implementing LDC. Over the 
past year, CRESST has been involved in revising, piloting, and validating the LDC Assignment Measure, 
based on findings from last year’s pilot. Our original measure consisted of eight dimensions, each 
targeting a discrete component of the module. Each dimension was scored on a four-point scale. Our 
current measure has nine dimensions, each scored on a 5-point scale. The revised version of the 
assignment measure focuses on rigor and content literacy, while also drawing more explicitly on the 
criteria for “what makes a great teaching task” and “what makes a great module” that are discussed in The 
1.0 Guidebook to LDC (See Appendix A for complete rubric). 
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The final measure includes only those areas for which we are able to collect sufficient evidence of 
classroom practice. The challenge with each of these potential domains of LDC implementation is the 
availability of information to support their assessment. The final rubric assumes that scorers have the 
following materials before them: 1) a completed template task (often printed from Module Creator); 2) 
one sample of supplemental instructional materials in reading instruction and one sample of instructional 
materials in writing instruction (e.g., graphic organizers, worksheets, lesson plans) that speak to the 
specificity of instruction; 3) three samples of student work on the template task, marked high, medium, 
and low; and 4) a short cover sheet where teachers indicate how long they spent teaching the modules and 
state their goals for content, reading, writing. Below, we make suggestions for additional classroom 
artifacts that might be collected in future iterations of the Assignment Measure to assist raters in making 
inferences about classroom instruction. 

Each dimension was scored on a five-point scale for scoring to reduce any clustering or bias 
towards the mean. A score of 1 indicates that a dimension is not present or realized in the assignment 
measure artifacts; a score of 3 suggests that a dimension is moderately present or realized; and a score of 
5 indicates that the dimension is fully realized. The first five dimensions in the revised measure address 
the question, “what makes a great teaching task?” These dimensions center on the teaching task rather 
than the module in its entirety. This approach aligns with the steps suggested for building an effective 
module, as outlined in The 1.0 Guidebook to LDC (cf. Crawford, Galiatsos, Lewis, & Ottesen, 201 1): 

1) Effective Writing Task considers the degree to which the teaching task makes effective use of 
the template task’s writing mode (i.e., argumentation or explanation); requires sustained writing 
and effective use of ideas and evidence to substantiate claims; and is feasible for most students to 
complete (i.e., appropriate for the grade-level and subject matter). Sources of information for 
evaluating this dimension include the teaching task template options, the teaching task, and 
summary information sections of Module Creator. 

2) Alignment to the CCSS and Local and State Literacy and Content Standards focuses on the 
extent to which teaching task addresses content essential to the discipline, as well as reading 
comprehension and writing standards informed by local and state standards. Evidence for 
assessing this dimension can be found in the CCSS and state standards provided in Module 
Creator, as well as the literacy and content standards that teachers include in the module. 

3) Text Alignment is the degree to which the assigned texts address teaching task content. The 
main sources of information for evaluating this and the following two dimensions are the reading 
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texts themselves, as well as the task, resources, and links listed in the Module Creator Handout or 
referenced in the student work. 

4) Text Appropriateness is the degree to which the teaching task includes reading texts that are 
accessible to most students (i.e., appropriate for the grade-level and subject matter). Rater 
materials included sample readings that represented 8th grade Lexile levels. Raters were 
encouraged to refer to these exemplar texts when assessing text appropriateness. 

5) Text Rigor is the degree to which the teaching task includes reading texts that use and develop 
academic understanding and vocabulary, and offer opportunities for multiple interpretations and 
higher-order thinking. 

The remaining four dimensions of the revised assignment measure address the question, “what 
makes a great module?” The dimensions capture the extent to which teachers engage in high quality 
instructional strategies, and address the four stages of instructional practice delineated in the LDC 
Instructional Ladder. These dimensions also provide an opportunity to assess the coherence and clarity of 
the module as a whole: 

6) Fidelity to LDC Module Instruction is the degree to which module instruction, activities, and 
the teaching task address each of the four stages of instructional practice (preparation for the task, 
reading process, transition to writing, writing process). The main sources of information include 
the instruction section of the Module Creator handout and any information provided about the 
distribution of activities and time spent on each of the four stages. 

7) Quality Instructional Strategies considers the degree to which the module provides clear 
instructional strategies aimed at helping students develop literacy skills and successfully complete 
the teaching task, and the degree to which module instruction and activities scaffold critical 
thinking and performance in a way that is meaningful within the context of a given field or 
subject-matter. In addition to the instruction section of Module Creator and completed teacher 
logs, evidence for this dimension can be found in classroom handouts and examples of student 
work. 

8) Coherence and Clarity of Module is the degree to which there is logical alignment between 
the teaching task and other module goals with readings, mini-tasks, and instructional strategies. 
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All classroom artifacts should be used to assess this and the final dimension, including materials 
and information listed or uploaded into Module Creator, classroom handouts, and examples of 
student work. 

9) Overall Impression is a holistic assessment of LDC Module. This dimension gives raters the 
opportunity to make an overall assessment of LDC implementation, and determine the extent to 
which a module contributes to student college readiness and development of advanced literacy 
skills. 


Methods 

LDC Modules were collected from 8 th grade social studies and science teachers who were part of 
a larger quasi-experimental study of LDC implementation. We asked that teachers submit data on two 
modules if possible, preferably in the fall and spring semesters. Most teachers taught at least two 
modules, but some taught only one. Although timing of module submission varied, in all cases Module 2 
reflected instruction that occurred later in the academic year. Each module captured teacher LDC 
instruction over the course of 2-4 weeks. As part of the larger study, teachers were also asked to 
complete an online log twice per week during each week of LDC instruction. These online logs were 
designed on the online survey engine Qualtrics and we requested that teachers submit modules and 
classroom artifacts online when they completed logging on a given module. In addition to the specific 
instructional materials listed above, teachers had an opportunity to submit any additional materials that 
they believed would help us understand their module instructional practice. We followed up with teachers 
who had completed logging but did not submit their materials. The final sample included 21 social 
teachers, who submitted 40 social studies modules, and 17 science teachers who submitted 29 science 
modules. 

We used a within-subject design to explore teacher variation in LDC implementation. The design 
allowed us to investigate variation in LDC implementation over time, as well as between subject areas 
(science and social studies) and between states (Pennsylvania and Kentucky). 

Pilot Study 

We piloted the assignment measure and rater training in April 2013 with two expert teachers, 
including one science and one social studies teacher. The four science and social studies modules used 
for the pilot session were selected from a pool of completed modules that were submitted to our research 
team from study participants in the early spring. These modules included all of the classroom materials 
that teachers were asked to submit, and represented the types of teaching task topics similar to what raters 
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encountered in the official rating session, including those on climate change, transportation of hazardous 
materials, the Electoral College, and the American Revolution. Both raters found the dimensions logical 
and intuitive, and the training in LDC, Module Creator, and the CCSS thorough and useful for their own 
practice. Their suggestions for how we might clarify the rubric and rater training protocol were minor 
and almost all were incorporated. We determined that it took approximately 45 - 60 minutes for a teacher 
to rate a module. 


Recruitment and Training 

We recruited raters for a week-long scoring session held on June 17-21, 2013. We generated a 
list of eligible and potentially interested teachers from local district leaders and experts in social studies 
and science education and we ultimately invited over 90 teachers to apply. Five social studies and four 
science teachers were selected from a total pool of over 70 applicants. Raters were offered $200 for each 
day of participation, in addition to breakfast, lunch and parking. The recruitment letter explained: 


UCLA's Center for Research on Evaluation, Standards, & Student Testing (CRESST) and the 
Gates Foundation are investigating an instructional intervention aligned to the Common Core 
State Standards (CCSS) in 8th grade science and social studies classrooms. We are looking for 
experienced teachers to review and score instructional materials collected in middle school 
science and social studies classrooms during the 2012-2013 school year. Rating sessions will 
take place from Monday, June 17, 2013 - Friday, June 21, 2013. 

Job Details: 

- Raters will received a daily stipend of $200 ($25/hour) 

- Breakfast, lunch, and parking will be provided 

- Sessions will take place at the UCLA campus from 9:00AM-5:00PM 

Job Qualifications: 

- Experienced teachers who currently or recently taught 8th grade science and social studies 

- May not have been a UC Employee within the last two years 

If you are interested in rating instructional materials or would like further information about our 
study, please fill out and return the attached form April 30, 2013. 


Rater training and calibration occupied the first day and a half of the weeklong session. The first 
morning was devoted to introducing raters to LDC and Module Creator, as well as to familiarizing them 
with the dimensions of the rubric. During the afternoon of the first day and the morning of the next, 
social studies and science teachers independently scored two anchor modules in their subject areas. Each 
group of teachers met with a subject area expert who had also scored the modules to discuss and calibrate 
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their ratings. The remainder of the week was devoted to rating modules. Each teacher rated 
approximately 24 modules. 

Generalizability Study Design 

Generalizability theory, or G theory, is a statistical framework for determining the 
reliability of measurements under specific conditions. The theory asserts that there are multiple 
sources of error, rather than a single error term as in classical reliability theory. Each source of 
potential error is considered a facet, and the goal of a G-study is to determine the amount of error 
caused by each facet and the interaction of facets. We conducted Generalizability studies to 
investigate the reliability of module ratings, with the goal of separating true teacher variation 
from other sources of measurement error, for example rater variation or variation in LDC 
implementation over time (Shavelson & Webb, 1991). We also conducted a decision study, or D 
study, to estimate how generalizability coefficients would change if different aspects of the study 
(e.g., number of raters, number of modules) were altered. Therefore, for each set of modules, we 
estimated two kinds of reliability coefficients: a generalizability coefficient (p) reflecting 
consistency in relative score interpretations (i.e., ra nk ordering) and a dependability coefficient 
(4>) for absolute interpretations (i.e., judging performance against set criteria or standards). In 
both cases, we estimated hypothetical scenarios that vary the number of modules collected per 
teacher and the number of raters. Finally, we conducted exploratory factor analyses to investigate 
the extent to which one or more dominant factors or traits underlie the correlation patterns 
observed among ratings on the nine rubric dimensions. 

Social studies and science modules were analyzed separately because raters exclusively scored 
modules in their subject area. A fully crossed design with all raters scoring all modules in a given subject 
area was not feasible due to time constraints, but most raters scores majority of the teacher artifacts. We 
analyzed all valid data by using the missing data option in SPSS, which enabled us to omit an observation 
when there were missing values in the independent effects. 

Qualitative Data 

In addition to the analyses described above, we conducted 20-30 minute interviews with 
individual raters during the final two days of week to inquire about their experience with the assignment 
measure and their initial reactions to teacher implementation of LDC. The nine raters also completed a 
short survey on Qualtrics that asked them to review their experience over the course of the week. The 
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survey was comprised of three sections. The first asked raters to “indicate how strongly you agree or 
disagree with the following statement as it applies to each rubric dimension:” 

1) I clearly understood this dimension and the aspect of LDC module instruction that it was intended 
to capture. 

2) The LDC module notebook provides sufficient evidence to judge this aspect of LDC module 
instruction. 

3) I am confident about the ratings I assigned in this dimension. 

In the second section, the raters were asked to indicate how useful each of the following components were 
“as a source of information forjudging each dimension:” one-page information form; module creator 
handout; other planning materials; reading/texts; reading supports; writing supports; samples of student 
work; other instructional materials. Finally, raters were asked to comment on what knowledge, 
experiences, or personality characteristics might make someone a good rater of LDC modules. 

Results 

Table 1 presents mean scores for all social studies and science modules across all dimensions. 
Several cautions must be considered in reviewing these findings. First, given the limited sample size, we 
must be careful of making generalizations about social studies or science LDC implementation. It is 
important to note that different groups of raters scored each set of modules, and thus the scores are 
incomparable. Moreover, it is conceivable that science raters may have been more lenient in their scoring 
for any number of reasons, including their lack of experience integrating literacy and content. 

However, we see that, in general, science modules received higher ratings across dimensions, 
with the exception of two dimensions. This may appear counter-intuitive: one would think that social 
studies teachers would have an easier time integrating writing instruction, given the subject matter. The 
mean scores may suggest science teachers were better able to integrate subject matter content and literacy 
within the framework of LDC instruction. Science teachers were most successful at following and 
elaborating upon the structure of the LDC module. Their modules not only contained the four “skill 
clusters” -Preparing for the Task, Reading Process, Transition to Writing, and Writing Process — but 
these skills clusters included mini-tasks that went beyond the default options provided on Module Creator. 
Science teachers were also able to find texts that aligned with the writing task, but these were not 
particularly rigorous, in that they did not necessarily promote multiple interpretations or higher-order 
thinking. 

Social studies modules scored higher than science modules on the Effective Writing Task (the 
linchpin of the module) and Text Rigor (the academic substance of the texts) suggesting that social 
studies teachers struggled less with integrating writing tasks into their content instruction, and more with 
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designing instruction and supporting student execution of the task. Of all the dimensions, social studies 
teachers were most successful at finding texts to align with the writing task. Again, it is possible that 
social studies raters were more exacting in their ratings, given their experience integrating literacy and 
content instruction. For the remainder of the report, we present results for social studies and science 
modules separately. 


Table 1: Mean Scores for Social Studies and Science Modules Across Dimensions (Scale 1-5) 


Dimensions 

Social Studies 

Science 

1. Effective Writing Task 

3.40 

3.05 

2. Alignment to Literacy and Content Standards 

2.42 

3.25 

3. Text Alignment 

3.44 

3.52 

4. Text Appropriateness 

3.24 

3.41 

5. Text Rigor 

3.35 

3.05 

6. Fidelity to LDC Module Instruction 

3.04 

3.85 

7. Quality Instructional Strategies 

2.85 

3.09 

8. Coherence and Clarity of Module 

2.82 

3.23 

8. Overall Impression 

2.73 

3.04 


Social Studies Modules 

Descriptive Statistics by Module and State: Social Studies 

Table 2 presents descriptive statistics for social studies by dimension separated by module. A 
few comparative observations can be made. First, surprisingly, the average rating for the second social 
studies module was lower than the first in almost all dimensions except Overall Impression (D9) and 
Effective Writing Task (Dl), where the improvement from the first to the second module was slight. It is 
unclear why social studies modules would have, on average, declined over the course of the year, and it is 
important to consider that ratings could reflect teacher fatigue (and a lack of willingness to upload 
relevant, supplementary instructional materials). The case for teacher fatigue may be further supported by 
the consistent means for Overall Impression and Effective Writing Task between Modules 1 and 2; these 
two dimensions rely less on the presence of extensive artifacts and more on raters’ gut sense of the 
intellectual value and rigor of the instruction. That the means on D 1 and D9 remained consistent suggests 
that module quality may not have varied as much as may appear at first glance. 


Table 2: Descriptive Statistics by Dimension and Module for Social Studies Teachers 
(N=40) 


Dimensions 


Module 1 Module 2 

(N=19) (N=21) 

M SD M SD 
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Effective Writing Task 

3.37 

1.08 

3.39 

1.18 

Alignment to Literacy and Content Standards 

2.54 

1.25 

2.22 

1.02 

Text Alignment 

3.49 

1.07 

3.37 

1.31 

Text Appropriateness 

3.32 

0.78 

3.15 

1.11 

Text Rigor 

3.51 

1.09 

3.17 

1.26 

Fidelity to LDC Module Instruction 

3.19 

1.04 

2.80 

1.18 

Quality Instructional Strategies 

2.88 

1.05 

2.79 

0.99 

Coherence and Clarity of Module 

2.83 

1.15 

2.77 

1.11 

Overall Impression 

2.70 

0.93 

2.74 

1.02 


Tables 3 present descriptive statistics for social studies modules by state. 1 Again, we must be 
cautious about drawing conclusions from these results, as differences may be a sign of pre-existing 
differences between teachers. Nonetheless, we see here that scores on social studies modules did not vary 
tremendously by state. Perhaps the biggest distinction is that teachers in IU13 scored considerably higher 
on Fidelity to LDC Module Instruction (D6). The higher score on this dimension likely reflects the fact 
that all teachers from IU 1 3 submitted modules designed on the online LDC platform, Module Creator. 
Approximately half of Kentucky social studies modules were submitted using an older paper template for 
LDC or without any template. If it was difficult for raters to identify the four skill clusters -Preparing for 
the Task, Reading Process, Transition to Writing, and Writing Process — the module could not receive a 
score of 3 in D6. However it is important to note that a module could earn a 3 on D6 by simply 
defaulting to mini-task options automatically provided in Module Creator for each skill-cluster. In other 
words, the average score of 3.35 in D6 for IU13 social studies modules does not necessarily suggest that 
those teachers elaborated or expanded on the default options provided for instruction. The overall mean 
across dimensions for IU13 social studies modules (Af= 3.03) is only marginally higher than that for 
Kentucky (M= 2.99). 


Table 3: Descriptive Statistics for Social Studies Modules by Dimension and State 

(N=40) 


Kentucky’ 


Dimensions 

IU13 (N=18) 

M SD 

(N=22) 

M SD 

Effective Writing Task 

3.38 

0.95 

3.38 

1.26 

Alignment to Literacy and Content Standards 

2.71 

0.99 

2.10 

1.19 

Text Alignment 

3.36 

1.09 

3.48 

1.28 

Text Appropriateness 

3.07 

0.86 

3.36 

1.03 

Text Rigor 

3.02 

1.03 

3.58 

1.25 

Fidelity to LDC Module Instruction 

3.35 

0.64 

2.70 

1.33 

Quality Instructional Strategies 

2.80 

0.87 

2.86 

1.13 


1 See Appendix B for comparisons with scores on five LDC modules designed at the district level in 
Hillsborough, FL. 
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Coherence and Clarity of Module 
Overall Impression 


2.80 0.85 2.80 

2.76 0.90 2.68 


1.31 

1.04 


Generalizability Study: Social Studies 

Table 4 presents the estimated variance components for a teacher by rater by module (t*r*m) 
generalizability analysis for social studies modules. Again, the goal here is to separate true teacher 
variation from other sources of measurement error, for example rater variation or variation in LDC 
implementation over time. This model partitions variance into seven components; each column represents 
a source of variation in module ratings on each of the rubric dimensions. In this particular G study, if we 
found high systematic variation between raters that might suggest that the ratings were not reliable. On 
the other hand, we might expect to find high variation across modules, because teachers may improve at 
LDC implementation over time. The main effects reflect true variance across teachers (o 2 t) and error 
variance across raters and modules (o 2 r, o 2 m); a residual term (o 2 trm,e) combines the t*r*m interaction 
and residual error unexplained in the model. Across the dimensions, the results are quite promising: we 
see virtually no variation across raters, and high variation between teachers. In particular, it is important 
to note the high teacher variance captured in D9: Overall Impression, a dimension that asked raters to 
assess the degree to which the module contributed to student college readiness and development of 
advanced literacy skills. This suggests that raters were able to assess the overall instructional potential of 
the modules, separate from the particularities of LDC implementation. 

For the three dimensions where we see lower variation between teachers (Dl: Effective Writing 
Task; D3: Text Alignment; and D4: Text Appropriateness) we nonetheless see that a large portion of the 
variance for those dimensions is captured in the teacher by module interaction effect. The o 2 tm interaction 
suggests that certain teachers’ scores on these dimensions varied between their first and second module. 

In other words, although we do not see high variation overall between first and second modules (o 2 m), it 
appears that differences between modules were tied to particular teachers. As discussed earlier, this 
variation may reflect inconsistencies in how teachers assembled and uploaded module materials, or it may 
reflect true variation in teacher implementation of LDC. In either case, this variation suggests that any 
single module may not accurately represent teacher LDC implementation. Finally, the residual error term 
(o 2 trm,e) may also reflect systematic rater inconsistency and other sources of error not captured in the 
design. 

Table 4: Generalizability Studies of Social Studies Module Ratings (t*r*m*tr*tm*rm)* 



Percenta 

ge of Total Variance (%) \ 

Dimension 

a2t 

o2r 

o2m 

a2tr 

o2tm 

o2rm 

o2trm,e 

Effective Writing Task 

5.6 

0.0 

0.0 

11.3 

* 

22.2 

6.0 

* 

55.0 

Alignment to Standards 



42.2 

3.1 

1.8 

0.0 

16.9 

2.5 



33.5 

Text Alignment 

15.1 

1.8 

0.0 

2.3 

* 

35.3 

0.0 

45.5 
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Text Appropriateness 

14.6 

2.1 

0.0 

3.2 

* 

23.0 

1.3 

* 

55.9 

Text Rigor 

* 

33.1 

0.0 

2.3 

0.0 

* 

24.5 

0.0 

* 

40.1 

Fidelity to LDC Module Instruction 

* 

49.4 

0.0 

3.7 

0.0 

* 

22.7 

0.0 

* 

24.2 

Quality Instructional Strategies 

* 

32.7 

0.0 

0.0 

12.6 

* 

21.8 

0.1 

* 

32.9 

Coherence and Clarity of Module 

* 

40.2 

0.0 

0.0 

15.5 

12.0 

4.5 

* 

27.8 

Overall Impression 

* 

34.8 

2.6 

0.0 

7.2 

17.1 

1.7 

* 

36.5 


"indicates large proportion of variance captured by this facet. 


Decision Study: Social Studies 

We also conducted decision studies to determine dependability estimates under hypothetical 
scenarios that varied the number of modules and raters. Dependability estimates provide information 
about the consistency of absolute performance (in this case, on a given dimension) independent of others’ 
performance, rather than consistency of relative standing. These findings are presented in Table 5. In 
social studies, with two modules, estimated dependability with 3 raters exceeds .60 for all dimensions 
except Effective Writing Task, Text Alignment, and Text Appropriateness. The estimates for all three 
dimensions are slightly improved with 3 modules, but still below .5. These are the same three dimensions 
for which we saw low teacher variance, which may suggest that the dimensions are simply not effective in 
discriminating among teachers. On the other hand, considering that the teacher by module variance is 
quite large for all three of these dimensions, it is possible that the low dependability estimates for these 
dimensions raises questions about how many modules might be necessary to capture true teacher variation. 


Table 5: Social Studies: Decision Studies of Module Ratings by Dimension (t*r*m Design) 



Dependability Coefficients 


2 Modules 

3 Modules 


Raters (Crossed) 

Raters (Crossed) 

Dimension 

2 

3 

2 

3 

Effective Writing Task 

0.149 

0.182 

0.194 

0.237 

Alignment to Literacy and Content Standards 

0.680* 

0.721* 

0.754* 

0.790* 

Text Alignment 

0.327 

0.362 

0.413 

0.453 

Text Appropriateness 

0.339 

0.390 

0.423 

0.480 

Text Rigor 

0.585 

0.622* 

0.679* 

0.712* 

Fidelity to LDC Module Instruction 

0.719* 

0.741* 

0.794* 

0.811* 

Quality Instructional Strategies 

0.562 

0.614* 

0.632* 

0.684* 

Coherence and Clarity of Module 

0.648* 

0.709* 

0.701* 

0.759* 

Overall Impression 

0.601* 

0.656* 

0.672* 

0.724* 


"indicates dependability estimates greater than .60. 


Factor Analysis: Social Studies 
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Table 6 presents the principal component solutions extracted from the average teacher scores for 
each dimension over raters and modules. Overall Impression (D9) was initially excluded to avoid 
artificial unidimensionality in the data, but later included when we determined that it did not skew the 
loadings. The result of the factor analysis for social studies module ratings is encouraging: all 9 
dimensions load heavily on one factor, and account for 63% of the variance in social studies ratings. This 
suggests a dominant factor or trait underlying the nine dimensions of LDC implementation measured in 
the Assignment Measure, and makes the case for the coherence and conceptual validity of the tool. It is 
interesting to note that two of the dimensions (Alignment to Literacy and Content Standards: D2 and 
Fidelity to LDC Implementation: D6) load equally well on a second factor. Because both of these 
dimensions require the rater to evaluate the module according to outside criteria (e.g., standards, LDC 
framework), it is possible that they point to a trait that is distinct from how the rater might go about 
evaluating classroom instruction. 


Table 6: Principal Component Analysis of Social Studies Modules (N=40) 



Com] 

ronent 

1 

2 

D1 mean 

.736 

-.191 

D2 mean 

.604 

.656 

D3 mean 

.803 

-.415 

D4 mean 

.827 

-.351 

D5 mean 

.768 

-.389 

D6 mean 

.664 

.599 

D7 mean 

.906 

.119 

D8 mean 

.948 

.142 

D9 mean 

.927 

.029 


Extraction Method: Principal Component 
Analysis. 


Science Modules 

Descriptive Statistics by Module and State: Science 

Table 7 presents descriptive statistics for science by dimension separated by module. We found 
that with the exception of D5: Text Rigor, average ratings improved across all dimensions between 
Modules 1 and 2. This trend reflects what we would expect as teachers become more experienced in 
implementing LDC instruction. 


Table 7: Descriptive Statistics by Dimension and Module for Science Teachers ( N=29 ) 
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Module 1 
(N=15) 


Module 2 
(N=14) 


Dimensions 

M 

SD 

M 

SD 

Effective Writing Task 

3.00 

1.20 

3.09 

1.14 

Alignment to Literacy and Content Standards 

2.98 

1.24 

3.45 

1.06 

Text Alignment 

3.46 

1.24 

3.55 

1.14 

Text Appropriateness 

3.30 

1.18 

3.53 

1.06 

Text Rigor 

3.06 

1.29 

3.00 

1.20 

Fidelity to LDC Module Instruction 

3.59 

0.94 

4.04 

0.88 

Quality Instructional Strategies 

3.04 

1.18 

3.09 

1.08 

Coherence and Clarity of Module 

3.17 

1.22 

3.23 

1.16 

Overall Impression 

2.85 

1.14 

3.19 

1.08 


Tables 8 present descriptive statistics for science modules by state. Science modules in Kentucky 
(M=3.44) were consistently higher across dimensions than those from IU13 (M=3.05). It is important 
that we refrain from overstating these mean differences - all fall well within the standard deviations. 
Furthermore, observed differences could be the result of pre-existing differences between teachers. 
Nonetheless, given the consistency of these differences, it is worth exploring and comparing the 
professional development offered to IU13 and Kentucky science teachers, to identify whether and/or how 
Kentucky teachers were able to develop stronger modules. 


Table 8: Descriptive Statistics for Science Modules by Dimension and State ( N=29 ) 

Kentucky > 


Dimensions 

IU13 (N- 

M 

=14) 

SD 

(N= 

M 

-15) 

SD 

Effective Writing Task 

2.80 

1.15 

3.27 

1.14 

Alignment to Literacy and Content Standards 

3.18 

1.11 

3.21 

1.24 

Text Alignment 

3.39 

1.10 

3.62 

1.27 

Text Appropriateness 

3.10 

1.14 

3.69 

1.04 

Text Rigor 

2.80 

1.08 

3.25 

1.36 

Fidelity > to LDC Module Instruction 

3.59 

0.79 

4.00 

1.03 

Quality Instructional Strategies 

2.92 

1.00 

3.19 

1.24 

Coherence and Clarity of Module 

2.92 

1.10 

3.46 

1.23 

Overall Impression 

2.76 

0.99 

3.25 

1.19 


Generalizabilitv Study: Science 

Table 9 presents the estimated variance components for a teacher by rater by module (t*r*m) 
generalizability analysis for science modules. This model partitions variance into seven components. The 
main effects reflect true variance across teachers (o 2 t) and error variance across raters and modules (o 2 r, 
o 2 m); a residual term (o 2 trm,e) combines the t*r*m interaction and residual error unexplained in the 
model. The results here resemble those found for social studies module ratings, with a few differences. 
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Although we again see virtually no variance across raters, we do see high variance in the teacher by rater 
interaction for two of the dimensions: Text Alignment (D3) and Text Rigor (D5). And although we see 
high variation between teachers for four of the dimensions (Dl, D6, D7, D9), the variance components for 
the remaining dimensions are fairly low. 

How do we interpret these results? First, it is important to note once again the high teacher 
variance captured in D9: Overall Impression, a dimension that asked raters to assess the degree to which 
the module contributed to student college readiness and development of advanced literacy skills. This 
suggests that raters were able to assess the overall instructional potential of the modules, separate from 
the particularities of LDC implementation. On the other hand, the high variance in the rater by teacher 
interaction (o 2 tr) signals inconsistencies in rater understanding or use of the scoring rubrics with different 
teachers. That we see high variation for this interaction in two of the dimensions dealing with text 
selection (D3 and D5), suggests that science raters were less clear — or possibly systematically 
disagreed — about what constituted a relevant and academically challenging text in science. 

As for the five dimensions for which we see low variation between teachers (D2: Alignment to 
Content and Literacy Standards; D3: Text Alignment; D4: Text Appropriateness; D5: Text Rigor; and 
D8: Coherence and Clarity), we nonetheless see that a large portion of the variance for those dimensions 
is captured by the teacher by module interaction effect. The o 2 tm interaction suggests that differences 
between modules were tied to particular teachers. As discussed earlier, this variation may reflect 
inconsistencies in how teachers assembled and uploaded module materials, or it may reflect true variation 
in teacher implementation of LDC. In either case, this variation suggests that any single module may not 
accurately represent teacher LDC implementation. Again, the variance captured by residual error term 
(o 2 trm,e) remains high for all but one dimension, likely reflecting systematic rater inconsistency and other 
sources of error not captured in the design. 


Table 9: Generalizability Studies of Science Module Ratings (t*r*m*tr*tm*rm) 



Percentage of Total Variance (%) 


Dimension 

a2t 

o2r 

o2m 

a2tr 

o2tm 

o2rm 

o2trm,e 

Effective Writing Task 

* 

24.8 

0.0 

0.0 

3.7 

6.8 

11.3 

* 

53.4 

Alignment to Standards 

17.6 

0.0 

3.1 

17.6 

* 

35.2 

4.0 

* 

22.5 

Text Alignment 

7.1 

0.2 

0.0 

* 

25.4 

* 

41.2 

2.2 

* 

23.9 

Text Appropriateness 

9.1 

8.0 

0.0 

18.7 

* 

25.8 

0.4 

* 

38.1 

Text Rigor 

12.6 

0.0 

0.0 

* 

28.7 

* 

40.0 

7.2 

11.5 

Fidelity to LDC Module Instruction 

* 

33.8 

14.0 

8.7 

2.3 

0.0 

0.6 

* 

40.6 

Quality Instructional Strategies 

* 

34.9 

0.0 

0.0 

15.3 

0.0 

6.8 

* 

43.0 

Coherence and Clarity of Module 

15.0 

0.0 

0.0 

6.4 

* 

21.0 

7.2 

* 

50.4 

Overall Impression 

* 

27.9 

0.0 

0.0 

17.1 

4.8 

14.2 

* 

36.0 


"indicates large proportion of variance captured by this facet. 
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Decision Study: Science 

We also conducted decision studies to determine dependability estimates under hypothetical 
scenarios that varied the number of modules and raters. These findings are presented in Table 10. In 
science, estimated dependability with 2 modules and 3 raters is quite low for five of the nine dimensions, 
but exceeds .60 in Effective Writing Task, Fidelity to LDC Instruction, Quality Instructional Practices, 
and Overall Impression. That these four dimensions all deal with the general sense of the module, rather 
than the particularities of text selection, text preparation, and standards alignment, might suggest that 
science raters could reliably discern teacher overall fidelity of implementation, but were less confident 
about their interpretation of the specific components of module design. None of the dimensions are 
substantially improved by adding another module to the model. These findings of the decision studies are 
considerably less robust than what we saw with social studies, which might be a consequence of the 
smaller sample size. However, it also may suggest that raters of science modules may need additional 
training in LDC, and perhaps a better understanding of what the effective integration of literacy 
instruction and science content looks like. 


Table 10: Science: Decision Studies of Module Ratings by Dimension (t*r*m Design) 



Dependability Coefficients 


2 Modules 

3 Modules 


Raters 

Raters (Crossed) 

Dimension 

2 

3 

2 

3 

Effective Writing Task 

0.536 

0.616* 

0.624 

0.698* 

Alignment to Literacy and Content Standards 

0.338 

0.375 

0.405 

0.450 

Text Alignment 

0.150 

0.174 

0.186 

0.219 

Text Appropriateness 

0.202 

0.244 

0.243 

0.294 

Text Rigor 

0.244 

0.278 

0.290 

0.335 

Fidelity to LDC Module Instruction 

0.597 

0.670* 

0.654 

0.724* 

Quality Instructional Strategies 

0.653 

0.723* 

0.687 

0.767* 

Coherence and Clarity of Module 

0.347 

0.402 

0.430 

0.491 

Overall Impression 

0.543 

0.629* 

0.601 

0.684* 


indicates dependability estimates greater than .60. 


Factor Analysis: Science 

Finally, Table 1 1 presents the principal component solutions extracted from the average scores 
for each dimension over raters and modules. Overall Impression (D9) was initially excluded to avoid 
artificial unidimensionality in the data, but later included when we determined that it did not skew the 
loadings. The result of the factor analysis for science module ratings is encouraging: all 9 dimensions 
load heavily on one factor, and account for 67% of the variance in social studies ratings. This suggests a 
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dominant factor or trait underlying the nine dimensions of LDC implementation measured in the 
Assignment Measure, and makes the case for the coherence and conceptual validity of the tool. 


Table 1 1 : Principal Component Analysis of Science Modules (N=29) 



Component 

1 

2 

D1 mean 

.706 

-.537 

D2 mean 

.658 

-.142 

D3 mean 

.896 

-.194 

D4 mean 

.833 

-.329 

D5 mean 

.919 

-.241 

D6 mean 

.551 

.759 

D7 mean 

.884 

.355 

D8 mean 

.877 

.335 

D9 mean 

.955 

.121 

Extraction Method: Principal Component 

Analysis. 


Qualitative Results 

The qualitative data collected from the rater interviews and the rater survey sheds light on how 
raters experienced the scoring session and suggests ways to potentially improve rater reliability in the 
future. One key finding that emerges from analysis of qualitative data is the sense that the artifacts 
collected and assembled in each module notebook were not necessarily sufficient to assess or make 
inferences about LDC instruction. Both science and social studies teachers found this to be especially 
true for dimensions 6, 7, and 8 (Fidelity to LDC Implementation, Quality Instructional Strategies, and 
Coherence and Clarity). When asked to assess the relative strength of each source of information, both 
social studies and science teachers found the actual Module Creator print-out not useful or only somewhat 
useful for rating most of the dimensions, with the exception of Fidelity to Implementation (D6) and 
Alignment to Standards (D2), two dimensions that required teachers to consult specific sections of the 
print-out. But many teachers did not find it particularly useful in rating the text-related dimensions, 
Instructional Quality, Coherence and Clarity, and the Overall Impression. 

By contrast, both groups of raters found the actual readings and texts extremely useful in 
assessing most of the dimensions of the rubric, especially, not surprisingly, the text-related dimensions. 
Social studies raters also found the texts useful for rating Quality Instructional Practices, Coherence and 
Clarity, and Overall Impression. Interestingly, science teachers reported that the texts were not useful in 
making such inferences. Likewise, whereas social studies raters found Reading Supports extremely 
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useful in rating almost all dimensions, science raters mostly found them useful in rating D6, D7, D8, and 
D9, but Not Useful for rating D1-D5. At the risk of making too much of this finding, it is possible that 
science raters were less comfortable drawing inferences about literacy instruction from the assembled 
artifacts than social studies raters. This finding is supported by our impression from the interviews that 
science teachers had less experience teaching literacy than social studies teachers. 

Raters suggested certain sources of information that would have made scoring easier. First, one 
rater suggested that in addition to final student work, teachers submit evidence of student progress in the 
form of revised drafts. Several raters wished they had had more evidence of teacher implementation - 
lesson plans, scaffolds, and actual readings, for those modules where teachers just listed readings on 
Module Creator. Raters also wanted much more information about school context, student demographics, 
and most importantly, the extent of professional development and support that teachers received in 
implementing LDC. Given the extent to which LDC departs from business-as-usual in content classrooms, 
raters felt that knowing the extent and quality of teacher training in the approach would have informed 
their evaluation. 

We also asked raters to give their impressions of LDC as an intervention geared to helping 
teachers integrate literacy and content instruction. The majority of raters indicated support and 
enthusiasm for the intervention, but highlighted (as mentioned above) what they saw as a tremendous 
need for professional development and coaching. One rater warned that LDC should not be seen as a 
“magic bullet.” Several raters questioned whether the stronger modules reflected the strength of LDC as 
an intervention or the instructional skills that the teacher brought to the model. Raters emphasized that 
the strong modules basically reflected good instruction, and one rater thought that in some modules “LDC 
gets in the way.” Science raters, in particular, highlighted that science teachers will need much more help 
in teaching reading and writing, beyond the LDC template. At the same time, science raters were quite 
enthusiastic about the actual topics that they saw in the modules, and several stated that the modules gave 
them ideas for their own practice. Social studies teachers were less enthusiastic about the actual modules 
they rated. Several commented on the lack of differentiation and the lack of instructional support that 
pushed students to consider multiple perspectives and read text critically. Others were frustrated that the 
rubric did not include a dimension to rate the quality of the actual content delivered in the module. For 
example, they found instances where student work that was marked ‘high’ by the teacher included glaring 
historical inaccuracies. 

Discussion 

Overall, we are buoyed by the findings discussed above and believe the CRESST Assignment 
Measure to be a promising assessment tool for gauging teacher success in integrating literacy and content 
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instruction. In both the social studies and science analyses, we found low rater variance and high teacher 
(or teacher by module) variation. These findings indicate that raters generally found the dimensions 
intuitive and aligned with the available sources of information. Moreover, the factor analyses indicate 
that all dimensions load on a single factor, making the case that the CRESST Assignment Measure 
effectively measures a coherent trait that might be understood to be LDC implementation, or perhaps 
more generally, instructional quality in the integration of literacy and content. These findings are 
especially promising given our limited data set and the myriad logistical challenges of artifact collection. 

At the same time, we can identity certain questions that are worthy of further investigation and 
consideration. First, how much instructional material is required for raters to make informed, reasonable 
inferences about LDC implementation? Raters would ideally have additional artifacts on which to base 
their judgments, however requiring teachers to submit additional artifacts raises logistical considerations 
in terms of efficiency and cost-effectiveness. Nonetheless, if the tool is to be used in future evaluations of 
LDC instruction, it is worth exploring whether additional samples of classroom instruction (e.g., 
classroom handouts, actual lesson plans, and samples of all student work on mini-tasks) would increase 
rater reliability. Second, in both the social studies and science analyses, we saw considerable teacher by 
module variation. Clearly, a single module is not sufficient to achieve a high dependability estimate; 
evidence from the decision study suggest that 3 modules may be sufficient in social studies, but perhaps 
not in science. With teachers struggling to understand how to implement LDC, perhaps it is not 
surprising that quality varies considerably across modules. If that’s the case, it may be unreasonable to 
expect that we can get a reliable teacher-level score with a small sample size of modules. This, too, needs 
to be explored if the CRESST Assignment Measure is to be used in further evaluation of LDC instruction. 

Third, we found that the results for the science module ratings to be slightly less robust than those 
for social studies. It is unclear whether this is a result of smaller sample size, or whether science raters, in 
general, were less experienced and less familiar with instruction that integrates literacy and content. We 
suspect the latter based on our qualitative data and if so, there are implications not only for rater training, 
but also for how to support science teachers implementing LDC. It is particularly telling that the less 
robust findings in the science module analyses tended to cluster around the text-related dimensions (D3, 
D4, D5). These findings might suggest that science teachers (and raters) need additional support in 
identifying and evaluating texts that can be used in LDC implementation. 

Lastly, there are some indications in both the quantitative and qualitative data that the text-related 
dimensions might not be sufficiently distinct, or sufficiently discriminating. In future iterations it is worth 
exploring whether they might be collapsed. We only caution that this not be done prematurely. The three 
dimensions are designed to capture distinct aspects of module design -the selection of texts that align 
with the content demands of the template task (D3); teacher attention to grade appropriate reading levels 
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(D4); and teacher attention to the disciplinary or academic rigor of the reading (D5). Admittedly, at first 
glance these distinctions may appear subtle. However, we believe that they capture some of the nuance 
involved in developing effective and teachable modules of LDC instruction. 

How one chooses to revise the CRESST Assignment Measure no doubt depends on how it will be 
used. We see potential for the tool both in future evaluations of the program, as well as in professional 
development. If used for professional development purposes, we believe that more nuanced dimensions 
(e.g., the text dimensions described above) can be useful in helping teachers hone the skills necessary in 
designing instruction around texts. We might even suggest adding additional dimensions that assess 
reading instruction separately from writing instruction. Clearly, such hair-splitting might not be 
necessary in a large-scale evaluation. In both cases, however, we see great potential for the tool in 
helping content teachers design quality instruction that integrates Common Core State Standards. 
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Appendix A 


Each dimension is scored on a 5-point scale ranging from "Fully Present or Realized" to "Not Present or 
Realized." 


Fully 

Sufficiently 

Moderately 

Barely 

Not 

Present or 

Present or 

Present or 

Present or 

Present or 

Realized 

Realized 

Realized 

Realized 

Realized 

5 

4 

3 

2 

1 


IMPORTANT: Descriptions are provided for three anchor points in the scale: 5 (Fully Present or Realized), 
3 (Moderately Present or Realized), and 1 (Not Present or Realized). Use the intermediate points in the 
scale (4 and 2) to rate assessment practice that lies between 5 and 3 and 3 and 1. 


Dimension 1: Effective Writing Task 

Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 
Dimension 3: Text Alignment 
Dimension 4: Text Appropriateness 
Dimension 5: Text Rigor 

Dimension 6: Fidelity to LDC Module Instruction 
Dimension 7: Quality Instructional Strategies 
Dimension 8: Coherence and Clarity of Module 
Dimension 9: Overall Impression 


Dimension 1: Effective Writing Task 

Definition: Degree to which teaching task makes effective use of the template task's writing mode (i.e., 
argumentation or explanation); requires sustained writing and effective use of ideas and evidence to substantiate 
claims; and is feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 

Main Sources of Information: 

Module Creator Handout (Task) 

- Read and evaluate the teaching task, student background/prior knowledge, and summary information. 

- Evaluate the difficulty or ease students may encounter trying to answer the question. 

- Compare module teaching task to teaching task template options. 

5. Fully Realized 

The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order thinking 
and writing, and are appropriate for the grade-level and subject 

matter. 

4. 


3. Moderately Present or Realized 

Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level 
and subject matter. 

2. 


1. Not Present or Realized 

Minimal evidence of an effort to identify explicit and clear teaching 
task and performance expectations that provide opportunities for 
critical thinTLEhg and are appropriate for the grade-level and subject 
matter. 






Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 

Definition: Extent to which module addresses content essential to the discipline, as well as reading comprehension 
and writing standards informed by local and state standards. 

Main Sources of Information: 

Module Creator Handout (Task) 

- Read and evaluate the standards included in the module. 

- Module should include ELA as well as subject matter CCSS/state standards. 

- Compare and contrast the standards the module includes with those that could have been included. 

- Particular attention to content standards (CCSS History/Social Studies, Science, and Technical Subjects); State 
Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 

5. Fully Realized 

Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the topic 
and teaching task. 

4. 


3. Moderately Present or Realized 

Module broadly addresses content essential to CCSS and local or state 
standards in science or social studies and reading comprehension and 
writing. Standards are sufficiently aligned to the topic and teaching 
task. 

2. 


1. Not Present or Realized 

Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 


Dimension 3: Text Alignment 

Definition: Degree to which assigned texts address teaching task content. 

Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Read and evaluate texts (hard copies or online). 

Student Work 

- References in student work. 

5. Fully Realized 

Assigned readings address the disciplinary content in science or social 
studies and give students the opportunity to gather information needed 
to complete the task. Readings are well aligned to the topic and 
teaching task, and provide students with well-balanced perspectives. 

4. 


3. Moderately Present or Realized 

Assigned readings mostly address the disciplinary content in science or 
social studies and give students some opportunities to gather 
information needed to complete the task. Readings are sufficiently 
aligned to the topic and teaching task, and provide students with 
moderately balanced perspectives. 

2. 


1. Not Present or Realized 

Minimal evidence that assigned readings address the disciplinary 
content in science or social studies and give students the opportunity to 
gather information needed to complete the task. Readings are poorly 
aligned to the topic and teaching task, and do not provide students with 
well-balanced perspectives. 
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Dimension 4: Text Appropriateness 

Definition: Degree to which teaching task includes reading texts that are accessible to most students (i.e., 
appropriate for the grade-level and subject matter). 

Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Read and evaluate texts (hard copies or online). 

Student Work 

- References in student work. 

Anchor Readings 

- Read for examples of appropriate reading levels for 8 th grade students. 

5. Fully Realized 

Assigned readings are highly accessible and appropriate for most 
students in 8 th grade social studies or science classrooms. Selection of 
readings addresses the needs of students with a range of literacy skills, 
including students who are above, at, or below grade level, and English 
Language Learners. 

4. 


3. Moderately Present or Realized 

Assigned readings are mostly accessible and appropriate for the 
majority of students in 8 th grade social studies or science classrooms. 
Selection of readings sufficiently addresses the needs of students with a 
range of literacy skills. 

2. 


1. Not Present or Realized 

Assigned readings are not accessible or appropriate for students in 8 th 
grade social studies or science classrooms. Selection of readings poorly 
addresses the needs of students with a range of literacy skills. 


Dimension 5: Text Rigor 

Definition: Degree to which teaching task includes reading texts that use and develop academic understanding and 
vocabulary, and offer opportunities for multiple interpretations and higher-order thinking. 

Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Identify list of selected articles/links. 

- Read and evaluate texts (hard copies or online). 

- Consider issues of source credibility. 

Student Work 

- References in student work. 

5. Fully Realized 

Assigned readings require students to engage in higher-order thinking, 
and develop a strong academic understanding and vocabulary in social 
studies or science. Readings afford a deep conceptual and contextual 
understanding of the teaching task and topic. Selection of readings 
includes a broad range of credible primary and secondary sources. 

4. 


3. Moderately Present or Realized 

Assigned readings require students to engage in some higher-order 
thinking, and develop an adequate academic understanding and 
vocabulary In social studies or science. Readings afford a sufficient 
conceptual and contextual understanding of the teaching task and 
topic. Selection of readings includes a moderate range of credible 
primary and secondary sources. 

2. 


1. Not Present or Realized 

Assigned readings require students to engage in little higher-order 
thinking, or develop an academic understanding and vocabulary in 
social studies or science. Readings afford a limited conceptual and 
contextual understanding of the teaching task and topic. Selection of 
readings inqjqdes few credible primary and secondary sources. 
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Dimension 6: Fidelity to LDC Module Instruction 

Definition: Degree to which module instruction, activities, and teaching task address each of the four stages of 
instructional practice (preparation for the task, reading process, transition to writing, writing process). 

Main Sources of Information: 

Module Creator Handout (Instruction) 

Information Sheet 

- Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 

5. Fully Realized 

The module instruction, activities, and teaching task reflect deliberate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect demonstrable effort to develop 
instructional scaffolding within and across each stage of instruction. 

4. 


3. Moderately Present or Realized 

The module instruction, activities, and teaching task reflect moderate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect sufficient effort to develop 
instructional scaffolding within and across each stage of instruction. 

2. 


1. Not Present or Realized 

The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect inadequate effort to develop 
instructional scaffolding within and across each stage of instruction. 


Dimension 7: Quality Instructional Strategies 

Definition: Degree to which the module provides clear instructional strategies aimed at helping students develop 
literacy skills and successfully complete the teaching task. And the degree to which module instruction and 
activities scaffold critical thinking and performance in a way that is meaningful within the context of a given field or 
subject-matter. 

Main Sources of Information: 

Module Creator Handout (Instruction) 

Classroom Handouts 

Student Work 

- Evaluate extent to which instructional strategies guide student learning in literacy and ability to complete the 
teaching task. 

- Evaluate extent to which the module activities scaffold critical thinking and student performance within the 
context of the subject matter at the core of the teaching task. 

5. Fully Realized 

Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking in 
social studies or science. There is explicit attention to helping students 
develop an accurate understanding of the topic and teaching task, and 
literacy skills necessary to successfully complete the writing task. 

4. 


3. Moderately Present or Realized 

Instructional strategies and activities are available to support adequate 
student learning and critical thinking in social studies or science. There 
is moderate attention to helping students develop an understanding of 
the topic and teaching task, and literacy skills necessary to complete 
the writing task. 

2. 


1. Not Present or Realized 

Limited instructional strategies and activities available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding of 
the topic and teaching task, or literacy skills necessary to complete the 
writing task. 
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Dimension 8: Coherence and Clarity of Module 

Definition: Degree to which there is logical alignment between the teaching task and other module goals with 
readings, mini-tasks, and instructional strategies. 

Main Sources of Information: 

Module Creator Handout 

Classroom Handouts 

Student Work 

5. Fully Realized 

Strong alignment between the teaching task and goals of the module, 
including the CCSS and local and state literacy and content standards, 
with the readings, mini-tasks, student work, and instructional 
strategies. 

4. 


3. Moderately Present or Realized 

Moderate alignment between the teaching task and goals of the 
module, including the CCSS and local and state literacy and content 
standards, with the readings, mini-tasks, student work, and 
instructional strategies. 

2. 


1. Not Present or Realized 

Poor alignment between the teaching task and goals of the module, 
including the CCSS and local and state literacy and content standards, 
with the readings, mini-tasks, student work, and instructional 
strategies. 


Dimension 9: Overall Impression 
Definition: Holistic assessment of LDC Module. 

Main Sources of Information: 

Module Creator Handout 
Classroom Handouts 
Student Work 

Main question: To what extent does this module contribute to student college readiness and development of 
advanced literacy skills? 

5. Advanced LDC Module Implementation 

4. Proficient LDC Module Implementation 

3. Adequate LDC Module Implementation 

2. Marginal LDC Module Implementation 

1. Inadequate LDC Module Implementation 
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Appendix B: Hillsborough Modules 


UCLA CRESST's study is also evaluating district-wide implementation of LDC in 6 th grade 
Advanced Reading classrooms in Hillsborough, FL. The implementation differed significantly from 
what occurred in IU13 and Kentucky for two reasons. First, the modules were designed by literacy 
experts at the district level, not by individual classroom teachers. Second, although the modules 
included template tasks that asked students to write about content (e.g., physical fitness, child 
labor), developers were not saddled with the additional burden of integrating subject-specific 
content standards. Nonetheless, comparison between subjects and between states might reveal 
whether these differences in implementation were evident in module quality as measured by the 
CRESST Assignment Measure. 

We recruited an experienced middle school teacher, certified in social studies and English 
Language Arts, to rate the Hillsborough modules using the CRESST Assignment Measure. In 
comparing the ratings, it is important to keep in mind two limitations: (1) because the Hillsborough 
modules were designed by district leaders and distributed to teachers as mandated curriculum, we 
were effectively in possession of all the instructional materials that constituted instruction of a 
particular module. Such detailed evidence of module implementation stands in contrast to samples 
we collected from IU13 and Kentucky; (2) whereas a few of the modules include models of student 
work, these are not marked high, medium, and low, and in any event, do not reflect an individual 
teacher's assessment of student achievement. 


Table 1A: Average Module Ratings by Subject Area 


Dimensions 

KY and IU- 
13: 8 th grade 
Social Studies 

KY and IU- 

13: 8 th 
grade 
Science 

Hillsborough: 

6 th grade 
Advanced 
Reading 

1. Effective Writing Task 

3.4 

3.05 

* 

3.75 

2. Alignment to Standards 

2.42 

3.25 

3.13 

3. Text Alignment 

3.44 

3.52 

* 

3.88 

4. Text Appropriateness 

3.24 

3.41 

3.38 

5. Text Rigor 

3.35 

3.05 

3.25 

6. Fidelity to LDC 

3.04 

3.85 

* 

4.00 

7. Quality Instructional Strategies 

2.85 

3.09 

* 

3.75 

8. Coherence and Clarity of Module 

2.82 

3.23 

* 

3.75 

8. Overall Impression 

2.73 

3.04 

* 

3.25 


Indicates higher means. 
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Table IB: Average Module Ratings by State 


Dimensions 

IU13 

KY 

FL 

1. Effective Writing Task 

3.09 

3.33 

* 

3.75 

2. Alignment to Standards 

2.95 

2.66 

* 

3.13 

3. Text Alignment 

3.38 

3.55 

* 

3.88 

4. Text Appropriateness 

3.09 

3.53 

3.38 

5. Text Rigor 

2.91 

3.42 

3.25 

6. Fidelity to LDC 

3.47 

3.35 

* 

4.00 

7. Quality Instructional Strategies 

2.86 

3.03 

* 

3.75 

8. Coherence and Clarity of Module 

2.82 

3.23 

* 

3.75 

8. Overall Impression 

2.73 

3.04 

* 

3.25 


Indicates higher means. 


We see in Tables 1A and IB that Hillsborough modules scored higher on almost all 
dimensions when compared to the modules collected from IU13 and KY. This finding should not be 
surprising. The modules were developed by literacy experts and included elaborate lesson plans 
with carefully scaffolded instruction in reading and writing. However, on three dimensions, the 
modules from FL were more or less the same as the others. Putting aside the score on Alignment to 
Standards (D2] in Table 1A, which was difficult to assess since the Hillsborough Modules were not 
required to address content standards, we see that the means for D4 and D5 were roughly the same 
across subject area and across states. This finding may suggest two distinct, but related, challenges 
about LDC implementation in ELA classrooms. First, in choosing topics that appear to have high 
potential for student engagement, developers may struggle to find "naturally occurring" texts that 
are appropriate and aligned with students' reading level (e.g., one module included dense articles 
produced by UNESCO on global child labor practices). Second, and relatedly, in an effort to find 
readable texts, developers may sacrifice substantive, academic rigor (e.g., many of the readings 
could be characterized as fluff articles about how to build an exercise routine or the dangers of 
texting). 

Despite the methodological limitations of the above comparison, it has crucial policy 
implications for any scale-up of LDC implementation. In particular, we see that module quality may 
be improved if materials are designed and disseminated by instructional experts at the district level. 
On the other hand, the challenge of selecting subject-specific texts that are both age-level 
appropriate and academically rigorous may persist even under such conditions. A more conclusive 
study might compare modules in the same subject area designed at the district level and by 
individual teachers, as a way of investigating the affordances of each approach. 
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Appendix C: 

Descriptive Analyses of Teacher Survey 


Table Cl 

Years of Teaching Experience 


Type of experience 

N 

Mean 

Std Dev 

Minimum 

Maximum 

Years of teaching 

55 

13.65 

9.92 

1 

40 

Years taught in current 
school 

55 

6.33 

4.71 

1 

21 

Years taught in current 
district 

55 

11.64 

8.58 

1 

39 


Table C2 

Teaching of Different Student Populations ( n = 

55) 




Student population 

n 

Yes (%) 

n 

No (%) 

ELL students 

45 

81.8 

10 

18.2 

Special education students 

40 

72.7 

15 

27.3 

Students reading or writing below grade 
level 

50 

90.9 

5 

9.1 

Students with advanced literacy levels 

53 

96.4 

2 

3.6 


Table C3 

Type of Participation in the LDC Initiative ( n = 55) 


Type of participation 

n 

LDC teachers (%) 

Required 

51 

92.7 

Voluntary 

4 

7.3 
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Table C4 


Modules Developed and Taught 


School year 

n 

Mean 

Std Dev 

Minimum 

Maximum 

Modules taught in 2011-12 

47 

3.60 

.901 

0 

5 

Modules developed in 2012-13 

55 

0.11 

.369 

0 

2 

Modules taught in 2012-13 

55 

4.31 

.960 

2 

7 


Table C5 

Teacher Attitudes Regarding Literacy Instruction in Content Area Classrooms 


Question 

n 

Mean 

Std Dev 

Teachers from all content areas should help students 
improve their reading and writing skills. 

55 

2.85 

0.52 

Science and social studies teachers do not have time to 
teach reading and writing. 

55 

0.62 

0.83 

Writing assignments can help my students develop a 
deeper understanding of important concepts. 

55 

2.75 

0.55 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C6 

Percent of Class Time Spent on the Different LDC Components 


Components 

n 

Mean 

Std Dev 

Minimum 

Maximum 

Preparation for task/introducing the 
module 

54 

15.04 

6.71 

5.00 

30.00 

Reading process 

54 

39.76 

13.56 

15.00 

75.00 

Transition to writing 

54 

19.06 

8.16 

4.00 

40.00 

Writing process 

54 

26.15 

8.87 

10.00 

50.00 


Note. Means represent percents. 
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Table C7 


Degree of Emphasis Placed on Different Reading Strategies during LDC Instruction 


Strategies 

n 

No 

emphasis 

(%) 

Little 

emphasis 

(%) 

Some 

emphasis 

(%) 

A great deal of 
emphasis (%) 

Independent reading research 

54 

0.0 

13.0 

42.6 

44.4 

Making predictions/ previewing 

54 

0.0 

9.3 

57.4 

33.3 

Summarizing important points 

54 

0.0 

1.9 

25.9 

72.2 

Note-taking/ annotation 

54 

1.9 

3.7 

37.0 

57.4 

Identifying/ defining vocabulary 

54 

1.9 

9.3 

61.1 

27.8 

Analyzing text structure 

54 

1.9 

11.1 

33.3 

53.7 

Interpreting information from 
graphical text 

54 

3.7 

9.3 

51.9 

35.2 

Distinguishing fact from opinion 

54 

0.0 

5.6 

33.3 

61.1 

Drawing conclusions from textual 
evidence 

54 

0.0 

1.9 

18.5 

79.6 

Citing textual evidence to support 
claims 

54 

0.0 

0.0 

9.3 

90.7 

Evaluating strength/ weakness of 
evidence 

54 

1.9 

5.6 

29.6 

63.0 

Comparing arguments in two or 
more texts 

54 

3.7 

9.3 

33.3 

53.7 

Examining author’s perspective/ 
bias 

54 

1.9 

11.1 

33.3 

53.7 

Examining rhetorical devices 

54 

9.3 

35.2 

42.6 

13.0 
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Table C8 


Degree of Emphasis Placed on Different Writing Strategies during LDC Instruction 


Strategies 

n 

No 

emphasis 

(%) 

Little 

emphasis 

(%) 

Some 

emphasis 

(%) 

A great deal of 
emphasis (%) 

Generating ideas for writing 

54 

0.0 

14.8 

38.9 

46.3 

Outlining 

54 

1.9 

14.8 

50.0 

33.3 

Writing/ text structure 

54 

1.9 

0.0 

35.2 

63.0 

Formulating a thesis statement 

54 

1.9 

1.9 

22.2 

74.1 

Formulating a counter-argument 

54 

5.6 

11.1 

29.6 

53.7 

Writing an introduction 

54 

0.0 

0.0 

25.9 

74.1 

Writing a conclusion 

54 

0.0 

1.9 

25.9 

72.2 

Writing a body paragraph 

54 

0.0 

0.0 

18.5 

81.5 

Using transitional words or 
phrases 

54 

0.0 

7.4 

35.2 

57.4 

Incorporating quotes/ evidence 

54 

1.9 

1.9 

11.1 

85.2 


Table C9 

Frequency of Use of Strategies to Assess Student Understanding 




Strategies 

n 

Never 

(%) 

Rarely 

(%) 

Sometimes 

(%) 

Often (%) 

Listened as students discussed reading 
or writing with peers 

54 

0.0 

5.6 

20.4 

74.1 

Asked students to provide feedback to 
each other 

54 

0.0 

0.0 

38.9 

61.1 

Circulated and reviewed student notes 
and work 

54 

0.0 

1.9 

20.4 

77.8 

Collected and reviewed student writing 
exercises 

54 

0.0 

3.7 

18.5 

77.8 

Asked students to answer oral 
questions 

54 

0.0 

1.9 

24.1 

74.1 

Reviewed student rough drafts 

54 

0.0 

5.6 

33.3 

61.1 

Asked certain students to present 
writing to class 

54 

5.6 

24.1 

37.0 

33.3 

Assigned a quiz 

54 

16.7 

40.7 

31.5 

11.1 

Graded student work 

54 

0.0 

0.0 

25.9 

74.1 

Exit slips 

54 

3.7 

14.8 

46.3 

35.2 
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Table CIO 


Frequency of Use of Strategies to Respond to Misunderstanding 


Strategies 

n 

Never (%) 

Rarely (%) 

Sometimes 

(%) 

Often (%) 

Held one-on-one conference with 
student 

54 

1.9 

11.1 

61.1 

25.9 

Asked peer to provide feedback or 
organized peer editing session 

54 

1.9 

5.6 

37.0 

55.6 

Stopped class and modeled strategy 

54 

1.9 

11.1 

38.9 

48.1 

Scheduled in-class workshop time 

54 

9.3 

13.0 

46.3 

31.5 

Wrote specific comments on student 
work 

54 

0.0 

3.7 

37.0 

59.3 

Offered student a hint or suggestion 

54 

0.0 

1.9 

40.7 

57.4 

Gave student more time to try again and 
self-correct 

54 

0.0 

1.9 

27.8 

70.4 

Gave student the answer 

54 

7.4 

50.0 

33.3 

9.3 

Graded student work 

54 

0.0 

1.9 

22.2 

75.9 

Re-taught lesson segment 

54 

1.9 

18.5 

64.8 

14.8 

Planned to review skill in later lessons 

54 

0.0 

14.8 

53.7 

31.5 

Assigned grammar exercises 

54 

29.6 

48.1 

18.5 

3.7 
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Table Cl 1 


School and District Support for LDC 


Question 

n 

Mean 

Std Dev 

School administrators ... 




Have a firm understanding of the LDC framework 

39 

1.72 

1.12 

Have made formative assessment a priority at my school 

47 

2.28 

0.83 

Encouraged me to participate in the LDC initiative 

52 

1.96 

1.10 

Provided me with feedback about my instruction of the 
module(s) 

50 

1.62 

1.23 

Expressed concerns that teaching modules is taking time 
away from other instructional priorities 

48 

0.40 

0.94 

Communicated how the LDC framework is aligned with other 
school initiatives 

44 

1.14 

1.21 

District administrators ... 




Support the LDC framework 

45 

2.69 

0.67 

Have a firm understanding of the LDC framework 

39 

2.26 

1.02 

Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 

= agree. 



Table C12 

Individuals who Visited Teachers’ Classrooms during LDC Module Instruction ( n = 

55) 


Individuals 

n 

Yes (%) 

n 


No (%) 

District or network LDC project lead 

22 

40.0 

33 


60.0 

Principal 

45 

81.8 

10 


18.2 

Instructional coach/department head 

43 

78.2 

12 


21.8 

Teacher colleague 

31 

56.4 

24 


43.6 

Table C13 






Teacher Participation in LDC Professional 

Development in 2012-13 School Year ( n 

= 54) 


District/Subject 

n 

Yes n 


No 


Hillsborough County, FL Advanced 
Reading 

42 

77.8 12 


22.2 
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Table C14 


Number of Professional Development Sessions Attended in 2012-13 School Year 


District/Subject 

n 

Mean 

Std Dev 

Minimum Maximum 

Hillsborough County, FL Advanced Reading 

42 

1.76 

0.79 

1.00 4.00 


Table C15 

Types and Perceived Effectiveness of Professional Development in which Teachers Participated ( n = 42) 

Type 

Participated 

If participated, perceived effectiveness 

N 

Yes (%) 

n 

Yes (%) 

n 

No (%) 

One-on-one classroom visits 

19 

45.2 

18 

94.7 

i 

5.3 

Coaching 

15 

35.7 

9 

100.0 

0 

0.0 

Webinars 

1 

2.4 

0 

0.0 

0 

0.0 

Small group meetings 

25 

59.5 

11 

91.7 

1 

8.3 

School-wide meetings 

14 

33.3 

6 

85.7 

1 

14.3 

District-wide meetings 

33 

78.6 

13 

81.3 

3 

18.8 

Cross-district meetings 

2 

4.8 

1 

100.0 

0 

0.0 


Note. Only teachers who indicated that they participated in a specific type of PD were asked about the 
effectiveness. 
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Table C16 


Content Included in LDC Professional Development Sessions in which Teachers Participated [n =56) 


Question 

n 

Yes (%) 

n 

No (%) 

Using LDC as a way to implement CCSS 

37 

88.1 

5 

11.9 

Building a teaching task 

35 

83.3 

7 

16.7 

Finding appropriate content materials 

28 

66.7 

14 

33.3 

Designing modules 

17 

40.5 

25 

59.5 

Using the instructional ladder 

38 

90.5 

4 

9.5 

Using mini-tasks to address reading and writing skills 

33 

78.6 

9 

21.4 

Providing students with feedback on their writing 

32 

76.2 

10 

23.8 

Scoring student work with LDC rubric 

36 

85.7 

6 

14.3 

Building modules with Module Creator 

11 

26.2 

31 

73.8 

Differentiating module instruction to meet student needs 

25 

59.5 

17 

40.5 

Implementing modules with special education students 

9 

21.4 

33 

78.6 

Implementing modules with ELL students 

10 

23.8 

32 

76.2 

Implementing modules with students who read/write 
below grade level 

11 

26.2 

31 

73.8 

Implementing modules with students with advanced 
literacy levels 

29 

69.0 

13 

31.0 


Table C17 

Regularly Scheduled Common Planning Time with Colleagues to Discuss LDC ( n = 54) 

District/Subject 

n 

Yes (%) n 

No (%) 

Hillsborough County, FL Advanced Reading 

35 

64.8 19 

35.2 
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Table C18 

Frequency of Formal and Informal Teacher Collaboration around LDC 


Frequency 

N 

Scheduled meetings (%) 

n 

Informal discussions 

(%) 

At least once a week 

14 

40.0 

31 

57.4 

Every other week 

7 

20.0 

8 

14.8 

Once a month 

9 

25.7 

5 

9.3 

Once per 

quarter/trimester/ semester 

3 

8.6 

5 

9.3 

Never 

2 

5.7 

5 

9.3 

Note. Scheduled meetings ( n = 

35), informal meetings ( n =54 ) 




Table C19 

Perceptions of Teacher Collaboration during LDC Implementation 



Question 

n 

Mean 

Std Dev 

I would describe my LDC colleagues as collaborative 

Collaboration with my LDC colleagues helps me ... 

54 

2.15 

1.02 

More effectively use the LDC framework 

54 

2.33 

0.93 

Better support student learning 

54 

2.39 

0.94 

Develop LDC modules 

54 

1.28 

1.19 

Teach LDC modules 

54 

2.28 

0.92 

Revise LDC modules 

54 

1.89 

1.16 

Use the LDC framework rubric 

54 

2.02 

1.12 

Use students’ products to inform my instruction 

54 

2.07 

1.11 

Provide helpful feedback to students about their 
writing 

54 

2.06 

1.14 

Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = 

agree somewhat, and 3 = agree. 
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Table C20 


Teacher Perceptions of Efficacy in Teaching LDC Modules 


Question 

n 

Mean 

Std Dev 

I knew what skills my students needed in order to complete 
the teaching task 

54 

2.57 

0.77 

I knew the type of mini-tasks to give my students to prepare 
them to complete the template task 

54 

2.56 

0.79 

I understood how to use the LDC instructional ladder 

54 

2.35 

0.91 

Based on the information collected from using the LDC 
modules, I adjusted my instruction to meet the needs of 
individual students 

54 

2.52 

0.77 

Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree 

somewhat, and 3 

= agree. 



Table C21 

Potential Barriers to Use of LDC Modules 




Barriers 

n 

Mean 

Std Dev 

1 had sufficient time to prepare to teach modules 

54 

2.04 

1.01 

I felt adequately prepared to effectively use modules 

54 

2.31 

0.89 

It is difficult to find the time to respond to student writing 

54 

2.06 

0.86 

I am unsure about how best to give productive feedback to student 
writing 

54 

0.91 

0.96 

Using the LDC modules takes too much time away from covering 
required curriculum topics 

54 

1.00 

1.10 

It is challenging for me to find content-rich reading materials at my 
students’ reading level 

54 

1.02 

1.07 

It is challenging for me to find the time to develop modules 

54 

2.04 

1.12 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C22 


Teacher Perceptions of Whether LDC Modules Helped Them Meet Instructional Goals (n = 55) 


Question 

n 

Yes (%) 

n 

No (%) 

Find effective strategies for teaching my subject content 

43 

78.2 

12 

21.8 

Learn new ways to include formative assessment in my 
classes 

43 

78.2 

12 

21.8 

Develop new ways to teach literacy skills in my content area 

44 

80.0 

11 

20.0 

Learn detailed information about students’ literacy 
strengths and weaknesses 

40 

72.7 

15 

27.3 

Provide students with more detailed feedback about their 
writing 

51 

92.7 

4 

7.3 

Implement the CCSS 

49 

89.1 

6 

10.9 

Increase the rigor of writing assessments 

52 

94.5 

3 

5.5 

Better engage students 

45 

81.8 

10 

18.2 


Table C23 

Teacher Perceptions of the Effectiveness of LDC 




Question 

n 

Mean 

Std Dev 

Improving students' literacy skills 

55 

2.47 

0.81 

Providing a curricular resource for teachers to address the 

CCSS 

55 

2.55 

0.74 

Encouraging science and social studies teachers to teach 
literacy skills 

55 

2.15 

0.97 

Encouraging secondary school teachers to teach literacy skills 

55 

2.42 

0.83 

Making instruction more engaging for the students. 

55 

2.24 

1.00 

Using formative assessment to identify student strengths and 
weaknesses to inform instruction 

55 

2.16 

1.00 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C24 


Student Engagement during LDC Module Instruction (n = 54) 


Effect on engagement 

n 

LDC teachers (%) 

More engaged 

29 

53.7 

Same level of engagement 

17 

31.5 

Less engaged 

8 

14.8 


Table C25 

Perceptions of Student Success on LDC Tasks 


Strategies 

n 

No success 

(%) 

Little 

success (%) 

Some 

success (%) 

A great deal of 
success (%) 

Reading mini-tasks built into the 
instructional ladder 

54 

1.9 

5.6 

38.9 

53.7 

Writing mini-tasks built into the 
instructional ladder 

54 

1.9 

3.7 

35.2 

59.3 

Final writing task 

54 

0.0 

1.9 

24.1 

74.1 


Table C26 

Teacher Perceptions on Student Impact of LDC Modules 

Student impact 

n 

Mean 

Std Dev 

Resulted in higher quality student writing 

54 

2.46 

0.794 

Supporting my students’ college-readiness 

54 

2.50 

0.863 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C27 

Teacher Perceptions on Student Impact during Most Recent LDC Module ( n = 54) 


Student impact 

n 

Yes (%) 

n 

No (%) 

Majority of students improved their understanding of 
content 

49 

90.7 

5 

9.3 

Majority of students improved their literacy skills 

46 

85.2 

8 

14.8 
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Appendix D: 

Descriptive Analyses of LDC Logs 


Table D1 


Average Percent of Time Spent on Different Classroom Activities 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Lecture on subject 
matter content 

55 

6.95 

5.07 

50 

7.89 

6.00 

105 

7.40 

5.53 

Mini-Lessons 

55 

5.12 

5.70 

50 

5.90 

6.74 

105 

5.49 

6.20 

Explicit strategy 
instruction 

55 

9.10 

5.62 

50 

12.51 

9.57 

105 

10.72 

7.91 

Whole-class discussion 

55 

11.21 

8.22 

50 

12.49 

9.79 

105 

11.82 

8.98 

Small group work 

55 

14.18 

9.24 

50 

12.89 

13.10 

105 

13.57 

11.21 

Pair/share 

55 

6.20 

7.93 

50 

6.92 

8.19 

105 

6.54 

8.03 

Independent 

reading/writing 

55 

36.06 

16.73 

50 

30.95 

16.76 

105 

33.62 

16.86 

Student presentations 

55 

2.73 

4.01 

50 

3.80 

6.12 

105 

3.24 

5.13 

Other 

55 

8.47 

9.63 

50 

6.66 

8.69 

105 

7.61 

9.19 

Note. Means represent percents. 









Table D2 










How Teachers Introduce the Modules 










Module one 


Module two 


Overall 



n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Overview and/or 
review of topic 

46 

1.38 

0.53 

42 

1.45 

0.54 

88 

1.41 

0.53 

Connect topic to 
students' existing 
knowledge 

47 

1.45 

0.46 

42 

1.48 

0.53 

89 

1.47 

0.49 

Overview of readings 

44 

0.72 

0.67 

37 

1.03 

0.67 

81 

0.86 

0.68 

Review writing prompt 

46 

0.97 

0.75 

37 

0.97 

0.64 

83 

0.97 

0.70 

Review success 
criteria and/or rubric 

45 

0.51 

0.66 

35 

0.55 

0.63 

80 

0.53 

0.65 

Other 

23 

1.06 

0.83 

19 

0.89 

0.98 

42 

0.98 

0.89 


Note. Scale is 0=not today, l=touched on briefly, 2=focus of instruction. 
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Table D3 


What Students do to Prepare for Module Instruction 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Listened as I explained 
task 

44 

1.33 

0.59 

40 

1.46 

0.54 

84 

1.39 

0.57 

Came up with 
questions about topic 

42 

1.03 

0.65 

37 

0.84 

0.67 

79 

0.94 

0.66 

Made predictions 
about topic 

44 

1.08 

0.63 

36 

0.80 

0.69 

80 

0.95 

0.67 

Came up with 
questions about 
writing task 

41 

0.50 

0.59 

34 

0.60 

0.70 

75 

0.55 

0.64 

Re-wrote task in their 
own words 

41 

0.42 

0.62 

32 

0.35 

0.69 

73 

0.39 

0.65 

Brainstormed possible 
answers to prompt 

41 

0.55 

0.65 

36 

0.75 

0.76 

77 

0.64 

0.71 

Made predictions 
about reading 

42 

0.80 

0.55 

38 

0.69 

0.68 

80 

0.75 

0.62 

Reviewed exemplars 
of student work 

41 

0.34 

0.56 

31 

0.12 

0.40 

72 

0.24 

0.51 

Completed planning 
sheet (e.g., graphic 
organizer) 

43 

1.25 

0.72 

34 

0.72 

0.84 

77 

1.02 

0.81 

Discussed important 
strategies needed to 
complete task 

43 

1.09 

0.69 

37 

1.06 

0.70 

80 

1.08 

0.69 


Note. Scale is 0=not today, l=touched on briefly, 2=focus of instruction. 
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Table D4 


Reading Skills Students Worked on the Day of the Log 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Independent reading/ 
research 

53 

1.33 

0.55 

44 

1.36 

0.59 

97 

1.34 

0.56 

Making predictions/ 
previewing 

53 

0.96 

0.53 

42 

0.81 

0.56 

95 

0.89 

0.55 

Summarizing 
important points 

54 

1.45 

0.44 

44 

1.31 

0.59 

98 

1.39 

0.51 

Note-taking/ 

annotation 

52 

1.13 

0.52 

43 

1.34 

0.61 

95 

1.22 

0.57 

Identifying/ defining 
vocabulary 

53 

0.88 

0.56 

45 

0.93 

0.62 

98 

0.90 

0.58 

Analyzing text 
structure (e.g., how 
part relates to whole) 

51 

0.74 

0.61 

46 

0.91 

0.74 

97 

0.82 

0.68 

Interpreting 
information from 
graphical text 

51 

0.65 

0.50 

41 

0.56 

0.66 

92 

0.61 

0.57 

Distinguishing fact 
from opinion 

48 

0.31 

0.44 

46 

1.19 

0.63 

94 

0.74 

0.70 

Drawing conclusions 
from textual evidence 

53 

1.15 

0.59 

42 

1.26 

0.66 

95 

1.20 

0.62 

Citing textual evidence 
to support claims 

51 

0.87 

0.60 

46 

1.30 

0.65 

97 

1.07 

0.66 

Evaluating strength/ 
weakness of evidence 

50 

0.38 

0.50 

45 

1.17 

0.69 

95 

0.76 

0.72 

Comparing arguments 
in two or more texts 

47 

0.25 

0.36 

43 

0.69 

0.76 

90 

0.46 

0.62 

Examining author's 
perspective/ bias 

50 

0.41 

0.58 

42 

0.79 

0.67 

92 

0.58 

0.64 

Examining rhetorical 
devices 

48 

0.12 

0.34 

37 

0.18 

0.49 

85 

0.15 

0.41 

Other 

37 

0.79 

0.88 

30 

0.55 

0.87 

67 

0.68 

0.88 


Note. Scale is 0=not today, l=touched on briefly, 2=focus of instruction. 
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Table D5 

Extent Teachers Relied on Different Strategies to Assess Student Understanding during Reading Component 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Listened as students 
discussed text with 
peers 

52 

1.31 

0.47 

44 

1.39 

0.56 

96 

1.35 

0.51 

Circulated and 
reviewed student notes 

52 

1.36 

0.50 

45 

1.53 

0.50 

97 

1.44 

0.50 

Reviewed peers' 
feedback 

47 

0.65 

0.64 

36 

0.74 

0.69 

83 

0.69 

0.66 

Collected and 
reviewed student 
written responses 
and/or graphic 
organizers 

49 

1.24 

0.55 

39 

1.30 

0.64 

88 

1.26 

0.59 

Asked students to 
answer oral questions 

51 

1.02 

0.55 

44 

1.24 

0.57 

95 

1.12 

0.57 

Listened to students 
thinking aloud while 
reading 

49 

0.73 

0.62 

37 

0.85 

0.77 

86 

0.78 

0.68 

Led whole-class 
discussion 

51 

1.05 

0.63 

42 

1.27 

0.57 

93 

1.15 

0.61 

Listened to student 
questions 

50 

1.02 

0.53 

37 

1.13 

0.57 

87 

1.07 

0.55 

Assigned a quiz 

46 

0.06 

0.17 

31 

0.21 

0.56 

77 

0.12 

0.38 

Graded student work 

47 

0.87 

0.67 

39 

0.97 

0.74 

86 

0.91 

0.70 

Exit slips 

47 

0.44 

0.59 

36 

0.71 

0.81 

83 

0.56 

0.70 


Note. Scale is 0=not at all, l=to some extent, 2=to a great extent. 
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Table D6 


Strategies Used when Teachers Discovered Student Misunderstandings about Reading 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

One-on-one 
conference to provide 
feedback 

53 

0.83 

0.55 

41 

1.02 

0.64 

94 

0.91 

0.60 

Asked peer to provide 
feedback 

53 

0.92 

0.53 

42 

1.06 

0.66 

95 

0.98 

0.59 

Stopped class and 
modeled strategy 

52 

0.60 

0.55 

41 

0.96 

0.66 

93 

0.76 

0.63 

Wrote specific 
comments on student 
work 

50 

0.41 

0.51 

35 

0.52 

0.65 

85 

0.46 

0.57 

Scheduled in-class 
workshop time 

48 

0.27 

0.47 

34 

0.28 

0.53 

82 

0.28 

0.49 

Devoted time in lesson 
for students to use 
feedback 

50 

0.75 

0.54 

41 

0.91 

0.71 

91 

0.82 

0.62 

Grouped students 
together on a "need" 
basis for targeted 
instruction 

49 

0.66 

0.58 

37 

0.57 

0.60 

86 

0.62 

0.59 

Offered student a hint 
or suggestion 

51 

0.89 

0.50 

41 

1.04 

0.51 

92 

0.96 

0.51 

Gave student the 

answer 

49 

0.33 

0.42 

36 

0.36 

0.61 

85 

0.34 

0.51 

Gave student more 
time to try again and 
self-correct 

51 

0.79 

0.53 

37 

0.99 

0.57 

88 

0.87 

0.55 

Graded student work 

48 

0.64 

0.56 

39 

0.86 

0.66 

87 

0.74 

0.61 

Re-taught lesson 
segment 

48 

0.29 

0.44 

33 

0.31 

0.44 

81 

0.30 

0.43 

Planned to review skill 
in future lessons 

49 

0.61 

0.58 

35 

0.70 

0.67 

84 

0.65 

0.62 

Other 

30 

0.33 

0.57 

22 

0.35 

0.71 

52 

0.34 

0.63 


Note. Scale is 0=not at all, l=to some extent, 2=to a great extent 
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Table D7 


Areas of Writing Students Worked on the Day of the Log 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Generating ideas for 
writing 

43 

1.02 

0.68 

32 

1.17 

0.75 

75 

1.08 

0.71 

Outlining 

42 

0.61 

0.57 

33 

1.10 

0.75 

75 

0.83 

0.69 

Writing/text structure 

45 

1.35 

0.64 

37 

1.39 

0.73 

82 

1.37 

0.67 

Formulating a thesis 
statement 

43 

0.73 

0.74 

40 

1.39 

0.68 

83 

1.05 

0.78 

Formulating a counter- 
argument 

38 

0.14 

0.37 

39 

1.36 

0.73 

77 

0.76 

0.84 

Writing an 
introduction 

45 

0.91 

0.76 

41 

1.39 

0.69 

86 

1.14 

0.76 

Writing a conclusion 

45 

0.85 

0.67 

43 

1.34 

0.66 

88 

1.09 

0.70 

Writing a body 
paragraph 

45 

1.23 

0.68 

41 

1.45 

0.63 

86 

1.34 

0.66 

Using transitional 
words or phrases 

44 

1.04 

0.65 

33 

1.10 

0.73 

77 

1.07 

0.69 

Incorporating 

quotes/evidence 

45 

1.38 

0.62 

38 

1.54 

0.60 

83 

1.45 

0.61 

Style/ word choice/ 
syntax 

42 

0.94 

0.67 

32 

1.11 

0.78 

74 

1.02 

0.72 

Grammar conventions 

44 

1.04 

0.67 

34 

1.14 

0.74 

78 

1.08 

0.70 


Note. Scale is 0=not today, l=touched on briefly, 2=focus of instruction. 
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Table D8 

Extent Teachers Relied on Different Strategies to Assess Student Understanding during Writing Component 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Listened as students 
discussed draft with 
peers 

38 

1.09 

0.67 

33 

1.04 

0.68 

71 

1.06 

0.67 

Asked students to 
provide feedback to 
each other 

37 

1.15 

0.64 

31 

1.11 

0.64 

68 

1.13 

0.64 

Observed and 
reviewed student work 

39 

1.56 

0.45 

34 

1.62 

0.58 

73 

1.59 

0.51 

Collected and 
reviewed student 
writing exercises 

37 

1.22 

0.77 

32 

1.09 

0.92 

69 

1.16 

0.84 

Asked students to 
answer oral questions 

35 

0.86 

0.64 

27 

0.86 

0.69 

62 

0.86 

0.66 

Reviewed student 
rough drafts 

36 

1.25 

0.67 

34 

1.34 

0.70 

70 

1.29 

0.68 

Asked certain students 
to present writing to 
class 

36 

0.54 

0.68 

28 

0.56 

0.73 

64 

0.55 

0.70 

Assigned a quiz 

32 

0.04 

0.14 

24 

0.06 

0.22 

56 

0.05 

0.18 

Graded student work 

37 

0.65 

0.71 

28 

0.61 

0.81 

65 

0.64 

0.75 

Exit slips 

33 

0.17 

0.43 

25 

0.27 

0.51 

58 

0.21 

0.46 


Note. Scale is 0=not at all, l=to some extent, 2=to a great extent. 
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Table D9 


Strategies Used when Teachers Discovered Student Misunderstandings about Writing 




Module one 


Module two 


Overall 


Activities 

n 

Mean 

Std Dev 

N 

Mean 

Std Dev 

n 

Mean 

Std Dev 

Organized peer-editing 
session 

39 

0.82 

0.66 

35 

0.93 

0.74 

74 

0.87 

0.70 

Scheduled in-class 
workshop time 

38 

0.71 

0.76 

27 

0.88 

0.82 

65 

0.78 

0.78 

Held one-on-one 
conference with 
student 

40 

1.19 

0.57 

35 

1.38 

0.65 

75 

1.28 

0.61 

Devoted time in lesson 
for students to use 
feedback 

37 

0.80 

0.66 

32 

1.11 

0.72 

69 

0.94 

0.70 

Grouped students 
together on "need" 
basis for targeted 
instruction 

34 

0.56 

0.59 

28 

0.78 

0.77 

62 

0.66 

0.68 

Modeled skill using 
my own writing 

37 

0.67 

0.58 

34 

1.15 

0.68 

71 

0.90 

0.67 

Demonstrated skill 
using student's writing 

38 

0.75 

0.58 

29 

0.98 

0.70 

67 

0.85 

0.64 

Provided grammar 
mini-lessons 

33 

0.25 

0.38 

26 

0.37 

0.61 

59 

0.30 

0.49 

Wrote specific 
comments on student 
work 

35 

0.68 

0.65 

29 

1.00 

0.82 

64 

0.82 

0.74 

Had student revisit 
readings 

39 

1.12 

0.71 

31 

1.34 

0.60 

70 

1.22 

0.67 

Offered student a hint 
or suggestion 

39 

1.06 

0.58 

35 

1.19 

0.59 

74 

1.12 

0.58 

Gave student time to 
try again and self- 
correct 

40 

1.20 

0.52 

35 

1.47 

0.62 

75 

1.33 

0.58 

Corrected student 
writing 

36 

0.88 

0.62 

30 

0.97 

0.76 

66 

0.92 

0.68 

Graded student work 

34 

0.43 

0.64 

29 

0.61 

0.80 

63 

0.52 

0.72 

Re-taught lesson 
segment 

31 

0.25 

0.47 

26 

0.43 

0.58 

57 

0.34 

0.53 

Planned to review skill 
in future lessons 

34 

0.48 

0.61 

26 

0.60 

0.73 

60 

0.53 

0.66 

Other 

20 

0.30 

0.66 

16 

0.28 

0.58 

36 

0.29 

0.61 


Note. Scale is 0=not at all, l=to some extent, 2=to a great extent 
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Appendix E: Quasi-Experimental Results 


Table El 

2012-13 LDC Effect Estimates on FCAT Reading, 3-Level Model (Students 
within Time within School) 


Fixed Effect 

Coefficient (S.E.) 

Level 1 Variables (Student) 



Female 

0.028 

(0.006)* 

White 

-0.002 

(0.014) 

Hispanic 

-0.041 

(0.015)* 

Black 

-0.152 

(0.015)* 

Asian 

0.125 

(0.022)* 

English speaker 

-0.041 

(0.020)* 

Spanish speaker 

-0.050 

(0.023)* 

Free/reduced price lunch 

-0.142 

(0.007)* 

Grade 5 FCAT Reading 

0.648 

(0.005)* 

Level 2 Variables (Year Compared to 2009-10) 



Year 2010-11 

0.032 

(0.030) 

Year 2011-12 

0.061 

(0.033) 

Year 2012-13 

0.027 

(0.035) 

Level 3 Variables (School) 



Treated first in 2010-11 

-0.140 

(0.127) 

Treated first in 2011-12 

-0.137 

(0.067)* 

LDC Interactions by Year 2010-11 



Treated first in 2010-11 

-0.184 

(0.058)* 

Treated first in 2011-12 

-0.116 

(0.040)* 

Mean Prior Grade 5 Student Achievement 

0.174 

(0.054)* 

LDC Interactions by Year 2011-12 



Treated first in 2010-11 

-0.175 

(0.084)* 

Treated first in 2011-12 

-0.090 

(0.046)* 

Mean Prior Grade 5 Student Achievement 

0.167 

(0.059)* 

LDC Interactions by Year 2012-13 



Treated first in 2010-11 

0.042 

(0.083) 

Treated first in 2011-12 

0.020 

(0.054) 

Mean Prior Grade 5 Student Achievement 

0.227 

(0.063)* 


Note: ^significant at p =.05 
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Table E2 

Prior Achievement and Demographic Characteristics of Eligible and Matched Treatment and Comparison 
Groups for FCAT Reading Outcome Analysis 



Eligible Sample 

Matched Sample 

Student Characteristics 

LDC 

0=5,548) 

Comparison 

0=14,523) 

LDC 

0=5,338) 

Comparison 

(n=9.241) 

Prior Achievement on FCAT Reading 

234.1 

234.1 

234.1 

237.6 

Female (%) 

51.4 

52.8 

51.4 

51.4 

White (%) 

48.4 

40.6 

49.6 

49.6 

Hispanic (%) 

25.6 

32.9 

25.7 

25.7 

Black (%) 

14.2 

20.0 

14.2 

14.2 

Asian (%) 

5.1 

3.2 

4.8 

4.8 

English Speaker (%) 

82.5 

72.6 

83.2 

83.2 

Spanish Speaker (%) 

14.1 

22.4 

13.7 

13.9 

Free/reduced price lunch (%) 

50.3 

55.6 

49.3 

49.3 
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Table E3 


2012-13 LDC Effect Estimates on FCAT Reading, 2-Level Model Including Phase 2 Schools 
(Began LDC in 2011-12) with Prior Effectiveness Data in 2010-11 


Fixed Effect 

Coefficient (S.E.) 

Level 1 Variables (Student) 

Female 

0.071 (0.032)* 

White 

-0.052 (0.044) 

Hispanic 

-0.030 (0.041) 

Black 

-0.145 (0.035)* 

Asian 

0.021 (0.075) 

English speaker 

-0.058 (0.077) 

Spanish speaker 

-0.101 (0.079) 

Free/reduced price lunch 

-0.109 (0.027)* 

Grade 5 FCAT reading 

0.606 (0.021)* 

Level 2 Variables (School) 

LDC Treatment 

-0.051 (0.048) 

Mean Prior Grade 5 Student Achievement 

0.174 (0.041)* 

Prior School Effectiveness (2010-11) 

0.218 (0.099)* 

LDC treatment by Student Characteristics 

Interactions 

Female 

0.073 (0.038) 

Free/reduced price lunch 

-0.049 (0.038) 

Grade 5 FCAT reading 

0.056 (0.025)* 


Note: ^significant at p =.05 
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Table E4 


2012-13 LDC Effect Estimates on FCAT Reading, 2-Level Model Including Phase 2 Schools 
(Began LDC in 2011-12) with and without Prior Effectiveness Data in 2010-11 


Fixed Effect 

Coefficient (S.E.) 

Level 1 Variables (Student) 

Female 

0.073 (0.029)* 

White 

-0.038 (0.040) 

Hispanic 

-0.050 (0.044) 

Black 

-0.128 (0.033)* 

Asian 

0.047 (0.069) 

English speaker 

-0.064 (0.069) 

Spanish speaker 

-0.101 (0.071) 

Free/reduced price lunch 

-0.114 (0.025)* 

Grade 5 FCAT Reading 

0.604 (0.019)* 

Level 2 Variables (School) 

LDC Treatment 

-0.033 (0.043) 

Mean Prior Grade 5 Student Achievement 

0.173 (0.037)* 

Prior School Effectiveness (2010-11) 

0.229 (0.098)* 

School Missing 2010-11 

-0.020 (0.030) 

LDC treatment by Student Characteristics Interactions 

Female 

0.061 (0.035) 

Free/reduced price lunch 

-0.050 (0.034) 

Grade 5 FCAT Reading 

0.054 (0.023)* 


Note: *significant at p =.05 
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Probability of Treatment Assignment, 2011-12 


Figure El 

Relationship between Assignment Variable (Fifth Grade Reading in 2010-11) and Probability of 
Assignment into Advanced Reading in 2011-12 



Standardized Fifth Grade FCAT Reading, 2010-11 
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Probability of Treatment Assignment, 2010-11 


Figure E2 

Relationship between Assignment Variable (Fifth Grade Reading in 2009-10) and Probability of 
Assignment into Advanced Reading in 2010-11 



Standardized Fifth Grade FCAT Reading, 2009-10 
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Probability of Treatment Assignment, 2009-10 


Figure E3 

Relationship between Assignment Variable (Fifth Grade Reading in 2008-09) and Probability of 
Assignment into Advanced Reading in 2009-10 



Standardized Fifth Grade FCAT Reading, 2008-09 
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Figure E4 

FCAT Regression Discontinuity Graph for 2011-12 Showing Relationship between Assignment Variable 
(Fifth Grade FCAT Reading in 2010-11) and the Outcome Variable (Sixth Grade FCAT Reading in 2011-12) 
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Standardized Fifth Grade FCAT Reading, 2010-11 
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Figure E5 

FCAT Regression Discontinuity Graph for 2 0 1 0- 1 1 Showing Relationship between Assignment Variable 
(Fifth Grade FCAT Reading in 2009-10) and the Outcome Variable (Sixth Grade FCAT Reading in 2010-11) 



Standardized Fifth Grade FCAT Reading, 2009-10 
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Figure E6 

FCAT Regression Discontinuity Graph for 2009-10 Showing Relationship between Assignment Variable 
(Fifth Grade FCAT Reading in 2008-09) and the Outcome Variable (Sixth Grade FCAT Reading in 2009-10) 
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Standardized Fifth Grade FCAT Reading, 2008-09 
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Probability of Basic Performance on District Writing in 2011-12 


Figure E7 


District Writing Regression Discontinuity Graph for 2011-12 Showing Relationship between Assignment 
Variable (Fifth Grade FCAT Reading in 2010-11) and the Outcome Variable (Probability of Basic 
Performance on District Writing in 2011-12) 



Standardized Fifth Grade FCAT Reading, 2010-11 


11 


Figure E8 

District Writing Regression Discontinuity Graph for 2010-11 Showing Relationship between Assignment 
Variable (Fifth Grade FCAT Reading in 2009-10) and the Outcome Variable (Probability of Basic 
Performance on District Writing in 2010-11) 
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Standardized Fifth Grade FCAT Reading, 2009-10 
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